Photographing device, photographing method, and playback method

ABSTRACT

A photographing device according to the present invention comprises: an imaging section that captures a subject image and outputs image data; a motion picture recording section that records a motion picture based on the image data; a parameter setting section that sets a photographing parameter to change a photographing state; and a parameter control section that automatically changes the photographing parameter during the recording of the motion picture regardless of the set photographing parameter.

This application is a divisional application of U.S. patent applicationSer. No. 12/815,082 (referred to as “the '082 application” andincorporated herein by reference), filed on Jun. 14, 2010, titled“PHOTOGRAPHING DEVICE, PHOTOGRAPHING METHOD, AND PLAYBACK METHOD,” andlisting Keiji KUNISHIGE, Satoshi MIYAZAKI, Masaomi TOMIZAWA, and OsamuNONAKA as inventors, the '082 application claiming benefit, under 35U.S.C. §119, to the filing date of prior Japanese Patent ApplicationsNos. 2009-141902, 2009-141903, 2009-141904 filed on Jun. 15, 2009. Theseapplications are expressly incorporated herein by reference. The scopeof the present invention is not limited to any requirements of thespecific embodiments described in the applications.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a photographing device, such as adigital camera, a video camera, and a mobile telephone with camera, andin detail, to a photographing device having a function of capturing astill image and motion picture, a photographing method, and a playbackmethod.

2. Description of the Related Art

When photographing is performed, photographing parameters, such as anaperture and zoom position, are switched generally. This method iseffective when capturing a still image while switching photographingparameters before photographing in a trial-and-error manner to confirmits effect.

On the other hand, when capturing a motion picture, if photographingparameters are switched during photographing in order to introducevariation in a monotonous screen, camera shake will occur during theoperation. Further, the entire process of the trial-and-error operationto switch photographing parameters is photographed. Hence, a generaluser is reluctant to switch photographing parameters during thephotographing of a motion picture. As a result, a monotonous motionpicture lacking interest is obtained.

Further, when capturing a motion picture, it is difficult to know whento start and stop photographing, and only a motion picture of anathletic meeting or wedding ceremony, which is a formalized event, iscaptured with a movie camera. Capturing a motion picture in such a caseis likely to result in a monotonous motion picture lacking interest,with a similar composition and exposure.

As described above, when capturing a motion picture, it is likely thatonly a monotonous motion picture lacking interest is obtained, and therehave been made various proposals to solve such a problem. For example,Japanese Unexamined Patent Publication No. 2003-110995 (disclosed onApr. 11, 2003) discloses an image processing device capable ofsequentially connecting a plurality of intervals specified by a user andimparting a transition effect at a connected part. According to thisimage processing device, it is possible to easily perform a high-leveledition of a motion picture desired by a user without the need to beaware of expertise.

Further, Japanese Patent No. 4240108 (disclosed on May 15, 2008)discloses an image sensor capable of detecting a change in facialexpression and of photographing at a timing intended by a user.According to this image sensor, it is possible to prevent the timing ofphotographing from being delayed and to ensure recording of a facialexpression desired by a user.

Furthermore, a movie camera capable of capturing a still image whilecapturing a motion picture is disclosed in the following prior art.Japanese Unexamined Patent Publication No. 4-331583 (disclosed on Nov.19, 1992) discloses that a strobe zoom mechanism is not used at the timeof strobe flash when capturing a still image while capturing a motionpicture but an image signal is amplified in an AGC circuit in accordancewith information about distance. Further, Japanese Patent No. 3673528(disclosed on Mar. 10, 1995) discloses a camera capable of easilyselecting the priority of still image photographing or the priority ofmotion picture photographing by providing a motion picture photographingswitch and a still image photographing switch.

SUMMARY OF THE INVENTION

An object of the present invention is to provide a photographing devicecapable of capturing and playing back a varied moving image, aphotographing method, and a playback method.

The photographing device according to the present invention comprises:an imaging section that captures a subject image and outputs image data;a motion picture recording section that records a motion picture basedon the image data; a parameter setting section that sets a photographingparameter to change a photographing state; and a parameter controlsection that automatically changes the photographing parameter duringthe recording of the motion picture regardless of the set photographingparameter.

The photographing device according to the present invention comprises:an imaging section that captures a subject image and outputs image data;a motion picture recording section that records a motion picture basedon the image data; a parameter setting section that sets a photographingparameter to change a photographing state; a composition changedetermining section that determines a change in composition in aphotographing screen; and a parameter control section that changes thephotographing parameter during the recording of the motion pictureregardless of the set photographing parameter when the compositionchange determining section determines that the composition is stable.

The photographing device according to the present invention comprises animaging section that captures a subject image and outputs image data; amotion picture recording section that records a motion picture based onthe image data; a parameter setting section that manually sets aphotographing parameter to change a photographing state; a facialexpression determining section that determines a change in facialexpression of a subject based on the image data; and a parameter controlsection that changes the photographing parameter during the recording ofthe motion picture regardless of the manually set photographingparameter in accordance with change in facial expression determined bythe determining section.

The photographing method according to the present invention comprisesthe steps of: capturing a subject image and outputting image data;recording the image data; and changing the photographing parameterduring the recording of the motion picture based on the image dataregardless of the manually set photographing parameter to change aphotographing state.

The photographing device according to the present invention comprises:an imaging section that captures a subject image and outputs image data;a parameter control section that controls a photographing parameter tochange a photographing state when acquiring the image data; a releasesection to instruct to capture a still image; and a photographingcontrol section that acquires the image data in accordance with theinstruction to capture a still image and subsequently, while changingthe photographing parameter by the parameter control section, acquiresand records the image data of continuous shooting.

The photographing device according to the present invention comprises: ascreen dividing section that divides a photographing screen into aplurality of regions in accordance with information about focusadjustment; a focus adjustment determining section that assigns a weightdetermined for each divided region in accordance with a screen featurein the photographing screen divided by the screen dividing section anddetermines an order of focus adjustment, from which divided region towhich divided region the focus should be adjusted, in the screen inaccordance with the result of the weighting; and a focus control sectionthat drives a photographic lens in accordance with the order of focusadjustment determined by the focus adjustment determining section.

The playback method according to the present invention comprises thesteps of: storing a still image and image data of continuous shootingfollowing the still image; rearranging the stored image data in adirection opposite to the direction of the points of time of continuousshooting; and successively playing back and displaying the still imageand the images of the continuous shooting rearranged by an imageprocessing section.

The recording medium that supplies a computer-readable program forcausing a computer to execute processing according to the presentinvention causes a computer to execute processing comprising the stepsof: storing a still image and image data of continuous shootingfollowing the still image; rearranging the stored image data in adirection opposite to the direction of points of time of the continuousshooting; and successively playing back and displaying the still imageand the images of the continuous shooting rearranged by an imageprocessing section.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an electrical configuration of acamera according to a first embodiment of the present invention.

FIGS. 2A to 2C are diagrams for illustrating how to capture a variedmoving image by changing the aperture while capturing a motion picturewith the camera according to the first embodiment of the presentinvention.

FIGS. 3A to 3C are diagrams for illustrating how to capture a variedmoving image by changing the focal length while capturing a motionpicture with the camera according to the first embodiment of the presentinvention.

FIG. 4 is a flowchart showing the operation of camera control in thefirst embodiment of the present invention.

FIG. 5 is a flowchart showing the operation of parameter automaticcontrol of the camera in the first embodiment of the present invention.

FIG. 6 is a flowchart showing the operation of parameter automaticcontrol of the camera in the first embodiment of the present invention.

FIGS. 7A to 7C are diagrams showing displays on a display section whenparameter automatic control is performed in a modified example of thecamera in the first embodiment of the present invention.

FIGS. 8A to 8E are diagrams for illustrating how to capture a variedmoving image with a camera according to a second embodiment of thepresent invention.

FIG. 9 is a flowchart showing the operation of capturing a motionpicture of a person with the camera in the second embodiment of thepresent invention.

FIG. 10 is a flowchart showing the operation of capturing a motionpicture of a person with the camera in the second embodiment of thepresent invention.

FIG. 11 is a flowchart showing the operation of photographing a personwith a camera in a third embodiment of the present invention.

FIG. 12 is a block diagram showing an electrical configuration of acamera according to a fourth embodiment of the present invention.

FIG. 13 is a block diagram showing details of an image processingsection of the camera according to the fourth embodiment of the presentinvention.

FIG. 14 is a diagram showing a still image and an image of a motionpicture to be continuously captured with the camera according to thefourth embodiment of the present invention.

FIG. 15 is a diagram showing the way to play back a still image and amotion picture continuously captured with the camera according to thefourth embodiment of the present invention.

FIG. 16 is a diagram showing the way to play back a still image and amotion picture continuously captured in the reverse direction with thecamera according to the fourth embodiment of the present invention.

FIG. 17 is a diagram showing the way the orientation of the camera ischanging after a still image is captured with the camera according tothe fourth embodiment of the present invention.

FIGS. 18A to 18F are diagrams showing the way to record and display thecenter part of the screen as a motion picture even when the orientationof the camera changes after a still image is captured with the cameraaccording to the fourth embodiment of the present invention.

FIG. 19 is a flowchart showing the operation of camera control of thecamera according to the fourth embodiment of the present invention.

FIG. 20 is a flowchart showing the operation of changing art parametersof the camera according to the fourth embodiment of the presentinvention.

FIG. 21 is a flowchart showing the operation of determining an exposureswitch step of the camera according to the fourth embodiment of thepresent invention.

FIG. 22 is a flowchart showing the operation of determining an exposureswitch direction step of the camera according to the fourth embodimentof the present invention.

FIGS. 23A and 23B are diagrams showing the way to play back and displaya still image and a motion picture continuously captured at the time ofphotographing with the camera according to the fourth embodiment of thepresent invention.

FIG. 24 is a block diagram showing an electrical configuration of acamera according to a fifth embodiment of the present invention.

FIGS. 25A and 25B are diagrams showing an image when photographing isperformed in an art motion picture mode with the camera according to thefifth embodiment of the present invention, particularly, FIG. 25Ashowing an image when photographing starts and FIG. 25B an image whenphotographing ends.

FIG. 26 is a diagram showing a positional relationship between cameraand subject with the camera according to the fifth embodiment of thepresent invention.

FIGS. 27A to 27C are diagrams showing focus adjustment and regiondivision with the camera according to the fifth embodiment of thepresent invention, particularly, FIG. 27A being a graph representing arelationship between distance to subject and contrast, FIG. 27B showingan image of a subject, and FIG. 27C being a diagram showing a dividedimage region.

FIG. 28 is a diagram showing a relationship of weighting to determine aZa region and a Zb region for performing a focus shift with the cameraaccording to the fifth embodiment of the present invention.

FIG. 29 is a flowchart showing the operation of an art motion picturemode of the camera according to the fifth embodiment of the presentinvention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, preferred embodiments will be described in accordance withthe drawings using a camera to which the present invention has beenapplied. FIG. 1 is a block diagram showing an electric circuit of acamera 10 according to a first embodiment of the present invention. Thecamera 10 is a digital camera and includes a control section 1, animaging section 2, a face detecting section 2 a, a parameter settingsection 3, a motion determining section 4, a recording section 6, aparameter control section 7, a display section 8, an operation section9, a flash ROM 12, etc.

The imaging section 2 includes a photographic lens (zoom lens) having azoom function, an exposure control section, such as a shutter andaperture, an image sensor, a drive and read circuit of the image sensor,etc., and converts a subject image formed by the photographic lens intoimage data by the image sensor and outputs the data.

The face detecting section 2 a detects whether or not a person's face isincluded in the subject image based on the image data output from theimaging section 2 and processed by an image processing section 5 withinthe control section 1, and when a face is included, detects theposition, size, etc., thereof and outputs the detection result to thecontrol section 1. The detection of a face is performed by extractingthe shade of parts, such as an eye, nose, and mouth, in a face and usinga pattern matching method etc.

The control section 1 controls the total sequence of the camera 10 inaccordance with programs stored in the flash ROM 12. The control section1 has a facial expression change extracting section 1 a, a compositionchange determining section 1 b, and the image processing section 5. Thefacial expression change extracting section 1 a extracts a change infacial expression based on the image data output from the imagingsection 2, the detection result of a face output from the face detectingsection 2 a, and the pattern of parts of a face, such as an eye, nose,and mouth, and data about the change in a facial expression stored in afacial expression change database (DB) 6 a, to be described later.

The composition change determining section 1 b determines whether or notthe composition of the screen and the way to hold the camera 10 change.When determining a change in the composition of the screen, thecomposition change determining section 1 b makes use of a motion vectoretc. that is obtained based on the image data output from the imagingsection 2. The way to hold the camera 10 is determined based on adetermination output of the motion determining section 4, to bedescribed later.

The image processing section 5 has a motion picture section 5 h and astill image section 5 i, takes in image data output from the imagingsection 2, performs various kinds of image processing, such as thinningprocessing, trimming processing, edge enhancement, color correction, andimage compression, and performs image processing for displaying a liveview of a motion picture, for recording a still image and a motionpicture in the recording section 6, for playing back and displaying astill image and a motion picture, etc. The image processing section 5has a circuit for still image and a circuit for motion picture inaccordance with processing, such as a still imagecompression/decompression circuit, a motion picturecompression/decompression circuit, etc., for compressing/decompressingimages. Further, the image processing section 5 extracts ahigh-frequency component of image data and determines a contrast of asubject. Information about this contrast is used for adjusting focus ofa photographic lens and also used for making a determination at the timeof automatic parameter control, to be described later.

The parameter setting section 3 manually sets photographing parametersand outputs the parameters to the control section 1 using a zoom buttonto change the focal length in the imaging section 2, an aperture valuesetting operation section for setting an aperture value, etc. It mayalso be possible to automatically set an aperture value etc. inaccordance with the result of exposure control.

The parameter control section 7 controls the imaging section 2 based onthe photographing parameters manually set or automatically set in theparameter setting section 3. By controlling the photographingparameters, it becomes possible to adjust the angle of view, blurredbackground (depth of field), etc. By changing these parameters, it ispossible to apply various effects to enjoy photographing. Further, theparameter control section 7 detects the currently set values, such asthe aperture value and focal length, and output the values to thecontrol section 1.

The motion determining section 4 has a hexaxial sensor etc. that detectsphysical quantities in six axes in total: an acceleration in thethree-dimensional space, that is, a triaxial acceleration; and arotation speed in the three-dimensional space, that is, a triaxialangular speed. It is also possible for the motion determining section 4to determine whether or not the way to hold the camera 10 has changed.It is possible to determine that the way to hold the camera 10 of aphotographer has not changed even when the image changes due to zoomingetc., and it can be known whether or not the photographer aims at thesame subject without changing the way to hold the camera withouterroneous operations.

The operation section 9 includes various operation members, such as astill image release button, a motion picture button, a power switch, aplayback mode setting button, a menu button, and an automatic modesetting button, and determines the operation state of these operationmembers and sends the determination result to the control section 1. Thecontrol section 1 described above controls photographing and playback ina predetermined sequence in accordance with the operation state of theoperation member.

Pressing the still image release button instructs to capture a stillimage and the motion picture button instructs to start capturing of amotion picture when pressed for the first time and instructs to end thecapturing of a motion picture when pressed again. The automatic modesetting button is an operation member to set the automatic mode in whichphotographing parameters are switched automatically in accordance withthe facial expression of a subject, as will be described later. When atouch panel is provided on the front surface of the display section 8,it may also be possible to set the automatic mode with the touch panel.

When instructed to photograph with the release button, the recordingsection 6 records the image data of a still image and a motion picturecaptured by the imaging section 2 and subjected to compressionprocessing by the image processing section 3. The facial expressionchange database (DB) 6 a within the recording section 6 is a databaseshowing a relationship between the shapes, positions, etc., of the partsof a face, such as an eye, nose, and mouth, and the facial expression.Based on the relationship, it is possible for the facial expressionchange extracting section 1 a to represent the facial expression of aperson, such as the degree of smile, by a numerical value. The degree ofsmile is represented by a numerical value based on the shape of the eyeand mouth, whether or not teeth are seen, etc. The flash ROM 12 is anelectrically rewritable memory and stores control programs to beexecuted in the control section 1, and data such as adjustment values. Amemory other than the flash ROM 12, for example, a mask ROM may beaccepted as long as it can store the control programs etc.

The display section 8 has a display section, such as a liquid crystalpanel and an organic EL panel, disposed on the backside etc. of thecamera 10 and performs a live view display before photographing, normalplay back display of a recorded image, display of camera information,etc. It is possible for a photographer to observe the live view displayto determine a composition and timing.

Next, the automatic change (automatic mode) of the photographingparameters during the display of a live view and photographing of amotion picture in the present embodiment will be described using FIGS.2A to 2C and FIGS. 3A to 3C. In the present embodiment, the aperture andfocal length are changed based on the facial expression, in particular,the degree of smile of a subject person 21.

FIGS. 2A to 2C show an example in which the aperture value is switchedto another and the depth of field is changed as photographing parametersin accordance with the facial expression, such as the degree of smile.The facial expression, such as the degree of smile, is detected by thefacial expression change extracting section 1 a and a determination ismade by representing the degree of opening of the mouth and eyes, howthe teeth are seen, etc., by numerical values. During the display of alive view and photographing of a motion picture, the captured image isdisplayed on the display section 8, and on a parameter change displaysection 8 a under the captured image the degree of smile and theaperture value at that time are displayed.

FIG. 2A shows a case where the person 21, who is the subject positionedin the center, is smiling with the degree of smile of 100% and in thiscase, it can be said that the person plays the leading role and thebackground plays only a supporting role, and therefore, the aperturevalue is set to F1.8 so that the background is blurred. In FIG. 2B, thedegree of smile is 50%, lower than that in FIG. 2A. In this case, thedegree of smile is not high, and therefore, the subject person 21 doesnot desire a photograph in which only the person 21 is outstanding.Hence, the aperture is set to F4.0 and the depth of field is madesomewhat deeper.

In FIG. 2C, the subject person 21 closes her eyes and the degree ofsmile is reduced to 20%, and it can be said that in this scene thesubject desires the background to be focused. Hence, the aperture valueis set to F8 so that the depth of field is made deeper to make clear thebackground.

In this manner, in the examples shown in FIGS. 2A to 2C, the facialexpression is detected and the camera 10 is controlled to reduce theaperture as the subject plays a less leading role so that the depth offield is made deeper and the attention of a viewer is directed also tothe background. Due to such automatic parameter control, it becomespossible to perform photographing in which a viewer can enjoy what kindof image is captured in the background and also enjoy representationthat the subject person 21 plays a leading role. In this case, althougha photographer does not need to manipulate the aperture each time, avaried photograph can be obtained.

FIGS. 3A to 3C show examples, in which as photographing parameters, thefocal lengths are switched and the angle of view is changed inaccordance with the degree of smile. FIG. 3A shows a case where thesubject person 21 positioned in the center smiles and in this case, theperson plays a leading role, and therefore, the face part may be aclose-up and the background plays only a supporting role, and therefore,a wide region of the background does not need to be photographed. Whenthe face is closed-up, the zoom lens is moved toward the side of longerfocal lengths, and therefore, there is an effect to blur the background.In the example in FIG. 3A, the degree of smile is 100%, and therefore,the focal length is set to 80 mm (equivalent to that of a 35 mm film),the face of the person 21 is closed-up, and the depth of field is madeshallower to blur the background.

In FIG. 3B, the degree of smile is 50%, lower than that in FIG. 3A. Inthis case, the degree of smile is not so high, and therefore, aphotograph in which only the subject person 21 is outstanding will notbe desired by the person 21. Hence, the focal length is set to 60 mm,that is, the person 21 is photographed in a rather wide angle and thedepth of field is made somewhat deeper. In FIG. 3C, the subject person21 closes her eyes and the degree of smile is reduced to 20%, andtherefore, it can be said that in this scene the subject person 21desires to call a viewer's attention to the background. Hence, the focallength is set to 40 mm, that is, the person 21 is photographed in a wideangle, the depth of field is made deeper, and thus the background ismade clear to see.

In this manner, in the examples in FIGS. 3A to 3C, the facial expressionis detected and as the person 21 plays a less leading role, the camera10 is controlled to set the focal length toward the wide side, so thatthe person 21 is photographed in a wide angle (the person 21 is madesmaller relatively), the depth of field is made deeper because the focallength is set toward the wide side, and the background also calls aviewer's attention.

By performing the automatic parameter control in this manner, in theexamples shown in FIGS. 3A to 3C also, it becomes possible to performphotographing in which a viewer can enjoy what kind of image is capturedin the background and also enjoy representation that the subject person21 plays a leading role. In this case, although a photographer does notneed to operate the zooming operation member each time, a variedphotograph can be obtained.

When the angle of view changes due to zooming, the composition changesection 1 b detects change in composition. When the change incomposition is detected, if the motion determining section 4 determinesthat the way to hold the camera 10 does not change and the direction ofthe same person 21 is photographed, it is recommended to perform theautomatic parameter control on the assumption that the composition isstable.

Next, the operation of the camera 10 in the present embodiment will bedescribed using the flowchart of camera control shown in FIG. 4. Thecontrol section 1 performs processing including a flowchart, to bedescribed later, in accordance with programs stored in the flash ROM 12.When the camera 10 is turned on and started, the flow shown in FIG. 4 isstarted. First, whether or not the mode is the photographing mode isdetermined (S1). The camera 10 comprises the photographing mode and theplayback mode and the state of the mode selected by a photographer isdetermined based on the operation state of the operation section 9.

When the mode is the photographing mode according to the result of thedetermination in step S1, next, zero is set to an automatic flag if itis a timing of the initial setting of the mode setting (S2). When theautomatic flag is zero, the mode is the automatic mode and when theautomatic mode is set, the photographing parameters, such as theaperture and the focal length, are controlled to switch in accordancewith the facial expression of the person 21 during the display of a liveview and photographing of a motion picture.

When zero is set to the automatic flag, next, an image is captured and adisplay is produced (S3). In this step, the image data captured by theimaging section 2 is subjected to image processing in the imageprocessing section 5 and a display of a live view is produced on thedisplay section 8 before capturing a moving image or a still image, andduring the photographing of a motion picture, a moving image during thephotographing as shown in FIGS. 2A to 2C and FIGS. 3A to 3C is displayedon the display section 8.

Subsequently, the face and the facial expression are detected (S4). Inthis step, the face detecting section 2 a detects whether the person 21is included in the image data, and when included, the facial expressionchange extracting section 1 a detects the facial expression of theperson 21.

When the face and the facial expression are detected, next, whether ornot the parameter manipulation is performed is determined (S5). Here,the parameter setting section 3 determines whether or not the manualsetting of the aperture value, zooming, etc., is performed. When theparameter manipulation is not performed according to the result of thedetermination, next, whether or not the automatic setting is performedis determined (S11).

In step 11, whether or not the automatic mode setting button within theoperation section 9 is operated, or whether or not the composition ofthe screen changes for a predetermined period of time is determined.Here, the predetermined period of time is set, for example, to one tothree seconds, in which a viewer begins to feel that the compositiondoes not change and is monotonous when viewing a motion picture. In thepresent embodiment, whether or not the automatic setting is performed isdetermined based on the operation state of the automatic mode settingbutton and the change in the composition of the screen, however, this isnot limited, and either one will do.

When the automatic setting is performed according to the result of thedetermination in step S11, zero is set to the automatic flag (S12). Asdescribed above, at the time of initialization, zero is set to theautomatic flag in step S2, however, when it is determined that theparameter manipulation is performed in step S5, one is set to theautomatic flag in step S7, to be described later. When one is set to theautomatic flag, zero is set to the automatic flag in step S12.

When the automatic setting is not performed according to the result ofthe determination in step S11, or when zero is set to the automatic flagin S12, next, whether or not the automatic flag is zero is determined(S13).

When the automatic flag is zero according to the result of thedetermination in step S13, the parameter automatic control is performed(S14). In the parameter automatic control, the switch control of thephotographing parameters, such as the aperture and the focal length, isperformed in accordance with the facial expression of the subject person21, as described in FIGS. 2A to 2C and FIGS. 3A to 3C. The detailedoperation of the parameter automatic control is described later usingFIG. 5 and FIG. 6. In general, when a live view is displayed, it isfrequent that the aperture is constant, such as the full aperture,however, in the present embodiment, it is made possible to confirm thechange in the depth of field by varying the aperture value even when alive view is displayed.

When the parameter manipulation is performed according to the result ofthe determination in step S5, the parameters are changed in accordancewith the manipulation (S6). In step S5, the manual setting of theaperture value, zooming etc., is performed, and therefore, in step S6,the aperture value, the focal length, etc., are changed in accordancewith the manual setting. Subsequently, one is set to the automatic flag(S7). When one is set to the automatic flag, the camera is controlledbased on the photographing parameters manually set by a photographer.

When the automatic flag is set to one, next, whether or not thephotographing is still image photographing is determined (S15). Here,whether or not the still image release button is pressed and operated isdetermined. When the photographing is still image photographingaccording to the result of the determination, still image photographingis performed (S16). In this step, the image data output from the imagingsection 2 is subjected to image processing and the image data of thestill image is recorded in the recording section 6. When still imagephotographing is performed, the process returns to step S1.

When the photographing is not still image photographing according to theresult of the determination in step S15, next, whether or not thephotographing is motion picture photographing is determined (S17). Here,the motion picture button is operated in the motion picturenot-photographing state is determined. When the motion picture button isoperated, a flag, not shown schematically, is set until the motionpicture button is operated again, and the determination result in stepS17 is Yes.

When the photographing is motion picture photographing according to theresult of the determination in step S17, the motion picture is recorded(S18). Here, the image data output from the imaging section 2 issubjected to image processing and the image data of the motion pictureis recorded in the recording section 6. During the motion picturephotographing, processing is repeated in the order of step S18→S1→S2→ .. . →S5→S11-> . . . →S15→S17→S18. Hence, when the automatic flag is setto zero, the parameter automatic control is performed in step S14, thephotographing parameters are changed in accordance with the facialexpression of the subject person 21, and the moving image at this timeis recorded in the recording section 6. When the photographing is notmotion picture photographing according to the result of thedetermination in step S17, the process returns to step S1.

When the mode is not the photographing mode according to the result ofthe determination in step S1, whether or not the photographing mode isterminated during the photographing is determined (S21). When the modeis not the photographing mode, the process advances to step S21, and inaddition to this, when the motion picture button is pressed and operatedagain during the motion picture photographing, the process also advanceto step S21. When the motion picture button is operated again during themotion picture photographing, it is determined that the photographingmode is terminated during the photographing. When the photographing modeis terminated according to the result of the determination, thephotographing is terminated (S22). Here, motion picture photographingtermination processing is performed.

When the termination processing in step S22 is performed, or when thephotographing mode is not terminated during the photographing accordingto the result of the determination in step S21, next, whether or not themode is the playback mode is determined (S23). Here, whether or not theplayback mode setting button of the operation section 9 is operated isdetermined. When the mode is not the playback mode according to theresult of the determination, the process returns to step S1.

On the other hand, when the mode is the playback mode according to theresult of the determination in step S23, the playback mode is performed(S24). In the playback mode, a list of the captured images recorded inthe recording section 6 is displayed on the display section 8 and anenlarged display of a thumbnail selected by a photographer is produced.When the playback mode is performed, the process returns to step S1.

As described above, in the flowchart of camera control in the presentembodiment, when the automatic mode is set, the photographingparameters, such as the aperture and the focal length, are automaticallyswitched in accordance with the facial expression of the subject. Hence,when a live view display is produced or a motion picture is recorded,the image changes in accordance with the facial expression and inparticular, when a motion picture is recorded, it is possible to capturea varied image.

The parameter automatic control is performed also during the display ofa live view, not limited to during the recording of a motion picture.That is, before capturing a motion picture starts, steps are performedin the order of step S1→ . . . →S5→S11→ . . . S13→S14→S15 NO→S17 NO→S1and in step S14, the parameter automatic control is performed and thephotographing parameters are changed in accordance with the facialexpression of the person 21.

Next, the details of the parameter automatic control in step S14 aredescribed using FIG. 5 and FIG. 6. When the flow of the parameterautomatic control is started, first, whether or not a face is detectedis determined (S31). The face is detected in step S4, and therefore,whether or not the parts of a face are included in the image data isdetermined based on the detection result.

When a face is detected according to the result of the determination instep S31, next, whether or not the position and direction of the cameraare the same for a predetermined period of time is determined (S32).Here, based on the determination results of the motion determiningsection 4 and the composition change determining section 5, whether ornot the camera 10 is photographing the same subject in the samedirection is determined. Here, the predetermined period of time isassumed to be, for example, about one to three seconds, in which aviewer begins to feel that the composition does not change and ismonotonous when viewing a motion picture.

When the position and direction of the camera are the same for apredetermined period of time according to the result of thedetermination in step S32, next, whether or not the motion of thebackground is small is determined (S33). Here, the composition changedetermining section 5 analyzes the image of the image data and makes thedetermination based on the motion of the background. This is becausewhen the background is in motion, the image is not monotonous and it isinappropriate to change the photographing parameters.

When the motion of the background is small according to the result ofthe determination in step S33, next, whether or not the contrast of thebackground is high is determined (S34). Here, the image processingsection 5 extracts a high-frequency component of the image datacorresponding to the background part and determines whether or not thecontrast is high based on the high-frequency component. When thecontrast of the background is higher than the contrast of the face partof the person 21, the background gives an impression of mess, andtherefore, the effect to change the aperture is significant.

When the background contrast is high according to the result of thedetermination in step S34, next, whether or not there is a room tochange the aperture is determined (S35). When the subject is dark, ifthe aperture is reduced, the image becomes dark and conversely, when thesubject is bright, it may be overexposed, and therefore, in this step,whether or not the exposure is allowable even if the aperture is changedis determined. This determination is made by acquiring information aboutthe aperture value from the parameter control section 7.

When there is a room to change the aperture according to the result ofthe determination in step S35, next, the aperture is changed inaccordance with the facial expression (S36). In this step, as describedusing FIGS. 2A to 2C, the parameter control section 7 changes theaperture value in accordance with the facial expression of the person21. When the aperture is changed or when the results of thedeterminations in steps S32 and S33 are No, the process returns to theoriginal flow.

When the background contrast is low according to the result of thedetermination in step S34, or when there is not a room to change theaperture according to the result of the determination in step S35, it isnot possible to switch the aperture to another, and therefore, thephotographing parameter is changed by zooming in step S41 and subsequentsteps.

First, whether or not the background pattern position changes isdetermined (S41). For the composition in which the background is plainor the contrast thereof is not low and the pattern varies from thesubject in the center of the screen toward the periphery, it iseffective to change the background by zooming. When the background isplain, such as that in a certificate photograph, the change in thescreen by zooming is monotonous and not interesting. Hence, in thisstep, the background patter is determined and whether or not the patternchanges depending on the position in the screen is determined. Only whenthere is a change, zooming is performed.

When there is a change in the position according to the result of thedetermination in step S41, next, whether or not the zoom can be changedis determined (S42). Here, whether or not the shortest focal length endor the longest focal length end of the zoom lens is reached isdetermined.

When the zoom can be changed according to the result of thedetermination in step S42, the zoom is changed in accordance with thefacial expression (S43). In this step, as described using FIGS. 3A to3C, the parameter control section 7 changes the focal length inaccordance with the facial expression of the person 21. When the zoom ischanged in accordance with the facial expression, or the result of thedetermination in step S41 or S42 is No, the process returns to theoriginal flow.

When a face is not detected according to the result of the determinationin step S31, the object is not a person because no face is detected. Inthis case, the object of photographing can be thought, for example, anaccessory, a small item, a dish, etc., and therefore, in step S51 andsubsequent steps (refer to FIG. 6), the angle of view and the apertureare changed as in the case of the photographing of a person so that avaried image can be captured.

When a face is not detected, next, whether the position remains the samefor a predetermined period of time is determined (S51). Here, thecomposition change extracting section 1 b analyzes the image data anddetermines whether the position of the subject can be changed. Here, thepredetermined period of time is set to, for example, about one to threeseconds, in which a viewer begins to feel that the composition does notchange and is monotonous when viewing a motion picture.

When the position remains the same for a predetermined period of timeaccording to the result of the determination in step S51, next, whetheror not the motion of the background is small is determined (S52),similarly as in step S33. This is because performing the parameterautomatic control is inappropriate when the background is in motion. Themotion of the background is determined in the composition changedetermining section 1 b.

When the motion of the background is small according to the result ofthe determination in step S52, next, whether or not the backgroundcontrast is high is determined (S53), similarly as in step S34. When thebackground contrast is high according to the result of thedetermination, next, whether or not there is a room to change theaperture is determined (S54), similarly as in step S35. When there is aroom to change the aperture according to the result of thedetermination, next, whether or not the room is toward the open side isdetermined (S55).

When there is a room toward the open side according to the result of thedetermination in step S55, next, the aperture is controlled toward theopen side (S56). Here, the parameter control section 7 gradually opensthe aperture in the imaging section 2 toward the open side. On the otherhand, when there is not a room toward the open side according to theresult of the determination in step S55, the aperture is controlledtoward the closed side (S57). Here, the parameter control section 7gradually reduces the aperture toward the fully-closed side.

When the background contrast is not high according to the result of thedetermination in step S53, the change is small even if the aperturevalue is changed because the background contrast is low, and therefore,a varied image is not obtained. Hence, the angle of view is changed instep S61 and subsequent steps. First, whether or not the backgroundpattern position changes is determined (S61), similarly as in step S41.

When the background pattern position changes according to the result ofthe determination in step S61, next, whether or not the zoom can bechanged toward the wide side is determined (S62). When there is a roomto change the zoom toward the wide side according to the result of thedetermination, next, zooming is performed toward the wide side (S63).Here, the parameter control section 7 gradually moves the focal lengthof the zoom lens toward the wide side.

On the other hand, when there is not a room to change the zoom towardthe wide side according to the result of the determination in step S62,next, zooming is performed toward the telephoto side (S64). Here, theparameter control section 7 gradually moves the focal length of the zoomlens toward the telephoto side.

When zooming is performed toward the telephoto side in step S64, or whenzooming is performed toward the wide side in step S63, or when theaperture is controlled toward the closed side in step S57, or when theaperture is controlled toward the open side in step S56, or when theresult of the determination in step S51 or S52 is No, the change ofparameters is terminated (S65) if there is an operation or motion, andthe process returns to the original flow. When the photographingparameters are changed in accordance with the facial expression, thefacial expression changes with time and it is possible to change thephotographing parameters in accordance with the change in the facialexpression. However, for other than a face, the change of parameters iscarried out limitlessly, and therefore, when the composition of asubject is changed, the subject moves, or a photographer performs anoperation, the change of parameters is terminated.

As described above, in the first embodiment of the present invention, amotion picture is prevented from becoming monotonous by changing themotion picture, such as blurring the background and zooming in thesubject, and thus, the change in the image is made dramatic. If a motionpicture is given an effect by gradually blurring the background orzooming in the subject, an image pleasant to view is obtained, however,when a user is absorbed in photographing, it is not possible for theuser to perform such an operation quickly. If a user tries to performthe operation forcedly, camera shake will occur. Hence, in the presentembodiment, such a kind of transition effect is automatically performedby the camera 10. However, if this is done automatically at all times,an unintended image will be obtained, and therefore, the camera 10 is sodesigned that such an operation is performed only when predeterminedconditions to cause such a transition effect are satisfied.

The predetermined conditions to cause the transition effect include acase where there is expected an effect when the aperture is opened, forexample, when there is a room for the position of the aperture and thedistance to the subject is small. In this case, it may also be possibleto add an image of the background. Another case is also included, wherethere is a room to change the zoom toward the wide side. It may also bepossible to change the zoom toward the telephoto side, however, in thiscase, the same effect may be obtained by trimming the image afterphotographing. Further, another case is also included, where there is achange in the facial expression of the subject. In this case, asdescribed in the present embodiment, it may also be possible to changethe effect of the background in accordance with the change in the facialexpression.

Next, a modified example of the first embodiment of the presentinvention will be described using FIGS. 7A to 7C. In the firstembodiment, when the composition or the subject remains the same, themoving image is monotonous and lacks attractive appeal, and therefore,the photographing parameters are switched and the transition effects areenjoyed. However, if such changes in the photographing parameters beginsuddenly in the automatic parameter mode, there is a case where suchchanges are against a photographer's intension. Hence, in the presentmodified example, changes are not begun suddenly, but an advice displayis produced on an advice display section 8 b in the display section 8.

FIGS. 7A to 7C show examples of advice displays. In the state in FIG.7A, the degree of smile of the person 21 is high, and therefore, thecontrol is performed to blur the background so that the person becomesoutstanding in the automatic parameter control. In the present modifiedexample, before the control to blur the background is started, “Blurbackground” is displayed on the advice display section 8 b and at thesame time, “Yes” used when a photographer permits the blurring of thebackground and “No” when not permitting are displayed as shown in FIG.7A. A touch panel is provided on the front surface of the displaysection 8, and thereby, the photographer can select by touching eitherbutton.

Blurring the background should not be performed simply because theaperture can be changed, but it should be determined whether the sceneis such one in which the background includes various objects as shown inFIG. 7A and blurring the background brings about the effect of aclose-up of the person. When the background is plain or includes amonotonous landscape, such an effect is not brought about and when aperson is within a predetermined distance and the composition is a bustshot or a close-up shot, and not only the background but also otherparts are blurred, such an advice display to blur the background is notproduced.

When a photographer permits to blur the background, the parametercontrol section 7 moves the aperture toward the open side to blur thebackground and at this time, the control state of the photographingparameter is displayed on the advice display section 8 b as “Backgroundis blurred” (refer to FIG. 7B). When the limit value of thephotographing parameter is reached, the photographing parameter cannotbe changed any more, and therefore, as shown in FIG. 7C, “Cancelblurring?” is displayed on the advice display section 8 b. At this time,“Yes” used to permit returning photographing parameter to the originalone (aperture value in the case of the figure) and “No” when notpermitting are displayed.

The configuration of the modified example of the first embodiment is thesame as that in FIG. 1 and what is required is only to add a displaystep and a determination step appropriately to the flowchart shown inFIG. 4 to FIG. 6, and therefore, its detailed description is omitted.

Next, a second embodiment of the present invention will be describedusing FIG. 8 to FIG. 10. In the first embodiment, one of thephotographing parameters, that is, the aperture or the focal length, iscontrolled in accordance with circumstances. In contrast to this, in thesecond embodiment, it is possible to obtain a more varied image bychanging both the aperture and the focal length. In the firstembodiment, a photographer instructs to start and end motion picturephotographing by operating the motion picture button and all of theimage data is recorded in the meantime. In contrast to this, in thesecond embodiment, a motion picture corresponding to the part of imagedata including the change in the facial expression and the best facialexpression is recorded automatically. The configuration in the secondembodiment is the same as the block diagram shown in FIG. 1 in the firstembodiment, and therefore, its detailed description is omitted.

FIGS. 8A to 8E show images in the present embodiment. In the presentembodiment also, the automatic parameter control is performed similarlyas in the first embodiment, and therefore, it is possible to obtain avaried moving image in which the background is blurred or a person isclosed-up only by holding and directing the camera 10 to the samesubject as shown in FIGS. 8A to 8E. In the present embodiment, theperson 21 in the image in FIG. 8E shows the best facial expression, andtherefore, only the moving image from when the photographing parameterbegins to change in FIG. 8B to when the best facial expression is shownis recorded.

The flowchart of motion picture photographing of a person that enablessuch moving image photographing is described using FIG. 9 and FIG. 10.In this flowchart, the general processing is omitted, such as thedetermination of the photographing mode in step S1 in the camera controlshown in FIG. 4. When the motion picture photographing of a person isstarted, the flow shown in FIG. 9 is started, and the change in thefacial expression is monitored first and the change is recorded (S70).Here, the facial expression change extracting section 1 a determines thefacial expression (degree of smile) of a person and temporarily recordsthe determination result in the recording section 6.

Subsequently, whether or not there is a change in the way to hold thecamera is determined (S71). Here, the composition determining section 1b determines whether or not there is a change in the way to hold thecamera 10 based on the output of the hexaxial sensor etc. within themotion determining section 4. The motion determination may be made notonly by the motion determining section 4 but also by the imageprocessing section 5 based on the change in image. When the way to holdthe camera does not change according to the result of the determination,next, whether or not zooming is on the way is determined (S72). There isa case where zooming is performed in step S95 (refer to FIG. 10), to bedescribed later, and in this step S72, whether or not zooming is inoperation is determined. Whether or not zooming is on the way isdetermined to avoid erroneous operations in the determination of thechange in composition in step S73.

When zooming is not on the way according to the result of thedetermination in step S72, that is, when zoom operation is not beingperformed, next, whether or not there is a change in composition isdetermined (S73). Whether or not there is a change in composition isdetermined by the composition change determining section 1 b based onthe image data.

When zooming is on the way according to the result of the determinationin step S72, next, whether or not there is only a change in image byzooming is determined (S74). Here, whether or not the change incomposition is only the change in image by the change in zoom isdetermined. Specifically, the composition change determining section 1 bacquires information about the focal length of the zoom lens from theparameter control section 7, performs image correction of the image databased on the information about focal length, and determines whether ornot there is a change in the corrected image.

When there is only a change in image by zooming according to the resultof the determination in step S74, or when there is not a change incomposition according to the result of the determination in step S73,next, the facial expression change extracting section 1 a determineswhether or not there is a change in facial expression (S75). When thereis a change in facial expression according to the result of thedetermination, photographing with a transition effect is performed(S78). Here, as described in FIG. 8, the aperture and the angle of vieware controlled based on the facial expression of the person 21. Detailedoperation of the photographing with a transition effect will bedescribed later using FIG. 10.

When the photographing with a transition effect is performed, next,whether or not the photographing ends is determined (S79). Thephotographing ends when the motion picture button of the operationsection 9 is operated again, and therefore, in this step, the operationstate of the motion picture button is determined.

When the photographing does not end according to the result of thedetermination in step S79, or when there is not a change in facialexpression according to the result of the determination in step S75, orwhen there is not a change in composition according to the result of thedetermination in step S73, or when there is not only a change in imageby zooming according the result of the determination in step S74, next,normal motion picture photographing is performed (S76). Here, the imagedata output from the imaging section 2 is subjected to image processingand then recorded in the recording section 6. It is of course possibleto record as a still image by pressing to operate the still imagerelease button at the timing after the normal motion picturephotographing in step S76. In this case, an image, in which the effectof the change in the photographing parameters is reflected, is displayedon the display section 8 and if there is an image that a photographer isinterested in, it is possible to perform photographing of a still imageat this timing.

Subsequently, whether or not the motion picture photographing is endedis determined (S77). Here, similarly as in step S79, the determinationis made based on the operation state of the motion picture button of theoperation section 9. When the photographing is not ended according tothe result of the determination, the process returns to step S70.Consequently, when the flow of the motion picture photographing of aperson is started, normal motion picture photographing is performedcontinuously in step S76, however, when a monotonous image results evenwhen there is a change in facial expression of a person, that is, theway to hold the camera remains unchanged, the composition is notchanged, but there is a change in facial expression of a person, it isdesigned so that the photographing with a transition effect is performedin step S78.

When the photographing is ended according to the result of thedetermination in step S77 or S79, next, whether or not the best facialexpression of a transition image is included is determined (S81). Instep S70, the change in facial expression, such as the degree of smile,is recorded and the photographing with a transition effect in step S78is performed, and therefore, in this step S81, whether or not the imageof the best facial expression is included in the image captured with atransition effect is determined.

When the best facial expression is included in the transition imageaccording to the result of the determination in step S81, next, imagesother than between the start of the transition to the end thereof aredeleted (S82). That is, the normal motion picture photographing isperformed, however, if the best facial expression is included in theimage captured with the transition effect when the facial expressionchanges, only the moving images in the interval during which thephotographing with the transition effect is performed are left andmoving images in other intervals are deleted.

Next, the photographing with a transition effect in step S78 will bedescribed using the flowchart shown in FIG. 10. When the flow of thephotographing with a transition effect is started, first, the startingpoint of time is recorded (S91). Here, information about date and timeoutput from the clock section within the camera 10, not shownschematically, is recorded. Subsequently, a display of instructing tofix the camera is produced (S92). Here, such a display is produced onthe display section 8 that instructs to fix the camera 10, preventingthe change in the direction of the camera 10.

When the display of instructing to fix the camera is produced, next,whether or not the aperture can be opened fully is determined (S93).Here, information about the current aperture value is acquired from theparameter control section 7 and compared with the fully open aperturevalue, and then, whether or not the aperture can be still opened towardthe fully open direction is determined. When the aperture can be fullyopened according to the result of the determination, photographing withfully open aperture is performed (S94). Here, the parameter controlsection 7 is instructed to drive the aperture fully open.

When the aperture cannot be fully opened according to the result of thedetermination in step S93, next, photographing toward the telephotodirection is performed (S95). Here, the parameter control section 7 isinstructed to drive the zoom lens toward the telephoto side. Asdescribed above, when the aperture can be driven fully open, theaperture is fully opened and when the aperture cannot be driven fullyopen, the zoom lens is driven toward the telephoto side.

When the photographing in the direction of fully open aperture in stepS94 is performed, or when the photographing in the telephoto directionin step S95 is performed, next, whether or not transition is completedis determined (S96). Here, whether the fully open aperture value isreached in the fully open aperture photographing or whether thetelephoto end is reached in the photographing in the telephoto directionis determined. When the transition is not completed according to theresult of the determination, the process returns to step S92.

On the other hand, when the transition is completed according to theresult of the determination in step S86, the completion point of time isrecorded (S97). The starting point of time is recorded in step S91 andthe completion point of time is recorded in step S97, and therefore, theinterval during which the transition photographing is performed isrecorded.

When the completion point of time is recorded, next, the photographingparameters are reset to the initial ones and photographing is continued(S96) and the process returns to the original flow. In steps S94 andS95, the aperture is driven to the fully open aperture value or the zoomlens is driven to the telephoto end, and therefore, in this step S96,the photographing parameters are reset to those before the drive and thephotographing is continued.

As described above, in the present embodiment, when the camera 10 isphotographing the motion picture of the same person, in order to preventan image from becoming monotonous, the photographing parameters arechanged and the photographing with a transition effect is performed whenthe facial expression of the person changes. Further when the bestfacial expression is included in the images captured by thephotographing with a transition effect, the images in the interval aresaved and the other images are deleted.

During the normal motion picture photographing, when the photographingwith a transition effect is performed a plurality of times, the imagesin the interval in which the facial expression is best are saved,however, this is not limited, and all of the images captured by thephotographing with a transition effect may be saved or the best two moreimages with the transition effect may be saved.

Next, a third embodiment of the present invention will be describedusing FIG. 11. In the first and second embodiments, a varied image iscaptured by changing the photographing parameters during thephotographing of a motion picture. In the third embodiment, a morevaried image is captured by capturing a motion picture while changingthe photographing parameters following the still image photographing.

In general, when photographing a still image, in most cases, the bestfacial expression of a subject is captured, and it may also be possibleto perform a motion picture photographing of the change in facialexpression when the subject relaxes after a still image photographingwith the transition effect of an image. In this case, it is convenientif a motion picture is captured automatically with the same operation asthat to photograph a still image without the need to set the motionpicture photographing purposely. Further, if a motion picture with thetransition effect is captured following a still image, it is possible tocreate a new motion picture by synthesizing the motion picture withanother and the synthesized motion picture can be made use of as a rawmaterial. It becomes also possible to represent the passage before astill image is captured by rewinding and playing back the motionpicture.

The configuration in the present embodiment is the same as the blockdiagram shown in FIG. 1 in the first embodiment, and therefore, itsdetailed explanation is omitted. The flowchart of photographing a personin the present embodiment is described using FIG. 11. In this flowchartalso, general processing, such as the determination of the photographingmode in step S1 in the camera control shown in FIG. 4, etc., is omitted.When the flow of photographing a person shown in FIG. 11 is started,first, whether or not still image photographing is performed isdetermined (S101). Here, the operation state of the still image releasebutton of the operation section 9 is determined.

When the still image release button is operated after letting timeelapse while waiting, the still image photographing is performed (S102).Here, the image data output from the imaging section 2 is subjected toimage processing by the image processing section 5 and the image data ofthe still image subjected to the processing is recorded in the recordingsection 6.

When the still image photographing is performed, next, whether or notonly one person is in the center is determined (S103) according to theresult of face detection by the face detecting section 2 a. When thereare two or more persons according to the result of the determination,this is not suitable for the photographing while changing thephotographing parameters, such as the angle of view and aperture, andtherefore, this flow is terminated.

On the other hand, when there is only one person according to the resultof the determination in step S103, next, whether or not the face is in aclose-up state is determined (S104). Here, whether not the size of theface detected by the face detecting section 2 a is in a close-up statewhere it occupies almost all the area of the screen is determined.

When the face is not in a close-up state according to the result of thedetermination in this step S104, next, motion picture photographing isperformed in a close-up with an overexposure (S121). Here, the motionpicture photographing is performed by instructing the parameter controlsection 7 to gradually drive the focal length toward the telephoto sideand at the same time, to gradually change the electronic shutter speed,the ISO sensitivity, etc., so that an overexposure is achieved. When thefocal length is near the telephoto end, it may also be possible toachieve a close-up by electronic zooming. Further, the photographingtime of the motion picture may be a predetermined period of time or thetime until the telephoto end is reached.

In step S121, achieving an overexposure may be omitted, or it may alsobe possible to perform photographing while changing other photographingparameters instead of achieving an overexposure, for example, whilechanging the focus position of the zoom lens. Some persons do not desiretheir faces to be photographed clearly, and in such a case, a close-upis omitted and the exposure and the focus may be changed considerably sothat their faces are unclear.

When the face is in a close-up state according to the result of thedetermination in step S104, next, whether or not the focal length can beshifted toward the wide angle side is determined (S105). Here, thedetermination is made by acquiring information about the current focallength from the parameter control section 7 and comparing it with thefocal length of the wide end.

When the focal length can be shifted toward the wide angle sideaccording to the result of the determination in step S105, next, motionpicture photographing is performed while shifting the focal lengthtoward the wide angle side (S106). Here, a motion picture is recordedbased on the image data output from the imaging section 2 whileinstructing the parameter control section 7 to gradually change thefocal length toward the wide angle side. The photographing time at thistime may be a predetermined period of time, or the time until the wideend is reached.

When the focal length cannot be shifted toward the wide angle sideaccording to the result of the determination in step S105, next, thecontrast distribution is determined (S111). Here, the image processingsection 5 extracts a high-frequency component of the image data anddetermines the contrast distribution.

When the contrast distribution determination is made, next, whether ornot the effect of aperture can be achieved is determined (S113). Whenthe current aperture value is near the fully open state, the depth offield can be made deeper by reducing the aperture and when the currentaperture value is near the smallest aperture, the depth of field can bemade shallower by opening the aperture toward the fully open side. Inthis step, whether or not the effect of aperture can be achieved isdetermined from the relationship between the current aperture value andthe contrast distribution.

When the effect of aperture can be achieved according to the result ofthe determination in step S112, motion picture photographing isperformed while changing the aperture (S113). Here, the parametercontrol section 7 is instructed to change the aperture value toward thedirection in which the effect of aperture can be achieved determined instep S112. The photographing time at this time may be a predeterminedperiod of time or may be the time until the fully open aperture value orthe smallest aperture value is reached. On the other hand, when theeffect of aperture cannot be achieved according to the result of thedetermination, the process advances to the above-described step S121 andphotographing is performed in a close-up with an overexposure.

When motion picture photographing is performed while changing theaperture in step S113, or when motion picture photographing is performedwhile shifting the focal length toward the wide angle side in step S106,or when motion picture photographing is performed in a close-up with anoverexposure in step S121, next, the still image and the motion pictureare associated with each other and then recorded (S122). Here, the imagedata of the still image captured in step S102 and the image data of themotion picture captured in step S106, S113, or S121 are associated witheach other. After the recording is completed, the flow of photographinga person ends.

As described above, in the third embodiment, during the still imagephotographing, a motion picture is captured while changing thephotographing parameters. Hence, it becomes possible to view not only amere still image but also a varied motion picture. Further, the stillimage and motion picture can be used as raw materials when editing theseimages to create a motion picture.

As described above, in each embodiment of the first to third inventions,during the recording of a motion picture based on image data, thephotographing parameters are changed in accordance with the motion of aphotographer and a subject regardless of the photographing parametersmanually or automatically set, and therefore, it is possible to capturea varied image in photographing.

Further, in the embodiments of the present invention, it is madepossible to capture a motion picture with changed photographingparameters regardless of the photographing parameters manually setfollowing the recording of a still image, and therefore, it is possibleto capture a varied image.

Furthermore, in the embodiments of the present invention, beforerecording a motion picture and when a live view is displayed, thephotographing parameters are changed regardless of the photographingparameters manually or automatically set, and therefore, it is possibleto confirm in advance whether or not a varied image can be captured byphotographing. When a live view is displayed, the aperture and the angleof view are changed, and therefore, it is possible to capture a stillimage when an image a photographer is interested in appears.

In the first to third embodiments of the present invention, as thephotographing parameters, the aperture value and the focal length (zoom)are described, however, these are not limited, and it may also bepossible to sequentially switch the focus positions or change theexposure (exposure correction value).

In the first to third embodiments of the present invention, the degreeof blurring as a photographing parameter is changed by controlling theaperture value, however, it may also be possible to change the degree ofblurring by performing image processing instead of the aperture when thebackground is blurred by image processing.

The facial expression change DB 6 a in the first to third embodiments ofthe present invention stores a database indicative of the relationshipbetween the shapes, positions, etc., of the parts of a face, such aseyes, nose, mouth, etc., and the facial expression of the face, however,it may also be possible to store data of speed when the facialexpression changes, in addition to the relationship. Further, in theembodiments, the photographing parameters are switched based on thefacial expression, such as the degree of smile, however, it may also bepossible to control the change of photographing parameters using thedata about the speed of change in facial expression while predicting thechange. Furthermore, in the embodiments, the facial expression isdetermined by the face detecting section 2 a as a motion of a subjectand the photographing parameters are changed based thereon, however,this is not limited and it may also be possible to change thephotographing parameters based on the motion of arm, motion of head,etc. Still furthermore, in the embodiments, the motion of a photographeris determined by the motion determining section 4 and whether or not thephotographing parameters are changed is determined based on the resultof the determination, however, it may also be possible to change thephotographing parameters directly in accordance with the motion of aphotographer.

Next, a fourth embodiment of the present invention will be describedusing FIG. 12 to FIG. 23. When a still image is captured and then amotion picture is captured for a predetermined period of time followingthe still image photographing, a varied image can be obtained.

FIG. 12 is a block diagram showing an electric circuit of the camera 10according to the fourth embodiment of the present invention. The camera10 is a digital camera and includes the control section 1, the imagingsection 2, the face detecting section 2 a, the parameter setting section3, the motion determining section 4, the recording section 6, theparameter control section 7, the display section 8, the operationsection 9, a clock section 11, the flash ROM 12, etc.

Like the imaging section 2 in the first embodiment, the imaging section2 includes a photographic lens (zoom lens) having a zoom function,exposure control sections, such as a shutter and aperture, an imagesensor, drive and read circuits of the image sensor, etc., and convertsa subject image formed by the photographic lens into image data by theimage sensor and outputs the image data. The photographic lens of theimaging section 2 includes a focus adjustment lens and its mechanism andin the fourth embodiment, it is also possible to capture a transitionmotion picture while shifting the focus. It may also be possible toprovide an aperture adjusting mechanism for blurring the background. Theface detecting section 2 a is the same as the face detecting section 2 ain the first embodiment, and therefore, its detailed description isomitted.

The control section 1 controls the total sequence of the camera 10 inaccordance with the programs stored in the flash ROM 12. The controlsection 1 has the composition change determining section 1 b, a trimmingsection 1 c, and the image processing section 5. The composition changedetermining section 1 b is the same as that in FIG. 1, and therefore,its detailed description is omitted. It may also be possible to providethe facial expression change extracting section 1 a shown in FIG. 1within the composition change determining section 1 b. It may also bepossible to design the facial expression change extracting section so asto predict the change in facial expression based on the speed of thechange when the facial expression changes in determining the change infacial expression.

The trimming section 1 c performs trimming processing to trim part ofimage data output from the imaging section 2. The trimming section 1 ctrims an image in the center of the screen when trimming image data,however, it may also be possible to design the trimming section 1 c soas to trim the periphery of the face of a subject based on the position,size, etc., of the face detected by the face detecting section 2 a, inaddition to the central image.

The image processing section 5 has a temporary recording section 5 a,the motion picture section 5 h, and the still image section 5 i andtakes in image data output from the imaging section 2, performs variouskinds of image processing, such as thinning processing, trimmingprocessing, edge enhancement, color correction, and image compression,and performs image processing for displaying a live view of a motionpicture, recording a still image and motion picture in the recordingsection 6, and playing back and displaying a still image and motionpicture. For image compression/decompression, the image processingsection 5 has circuits for a still image and motion picture inaccordance with processing, such as a still imagecompression/decompression circuit and a motion picturecompression/decompression circuit. It is also possible for the imageprocessing section 5 to acquire a motion vector based on image data. Aswill be described later, the temporary recording section 5 a is a memoryfor temporarily recording image data of a motion picture capturedfollowing the still image photographing. Details of the image processingsection 5 will be described later using FIG. 13.

The operation section 9 includes various operation members, such as arelease button, a power switch, a playback mode setting button, a menubutton, and an art mode button, and determines the operation state ofthese operation members and sends the determination result to thecontrol section 1. The control section 1 described above controlsphotographing and playback in a predetermined sequence in accordancewith the operation state of the operation members. The art mode buttonis an operation member to set the art mode, and when the art mode isset, a motion picture is captured (continuous shooting) while changingthe photographing parameters following the still image photographing.The art mode may be set from a menu screen that is displayed on thedisplay section 8 by the menu button without the need to provide the artmode button. Alternatively, it may also be possible to design so thatthe art mode is always set at the time of still image release withoutproviding the art button.

The display section 8 has a display section, such as a liquid crystalpanel and an organic EL panel, disposed on the backside etc. of thecamera 10, like the display section 8 in the first embodiment, andproduces a live view display before photographing, a normal playbackdisplay of a recorded image, a camera information display, etc. It ispossible for a photographer to observe the live view display todetermine a composition and timing. Further, the display section 8 inthe present embodiment produces a playback display of an image capturedin the art mode in which a motion picture is captured following thestill image photographing.

The clock section 7 performs a timing operation and outputs informationabout the date and time of photographing. The information about the dataand time of photographing is recorded at the same time when image dataof a still image and motion picture is recorded in the recording section6. When photographing in the art mode, the information is used todetermine whether or not a predetermined period of time elapses afterthe photographing of a motion picture (continuous shooting) is startedfollowing the still image photographing.

The parameter setting section 3, the determining section 4, therecording section 6, and the flash ROM 12 are the same as the parametersetting section 3 etc. in the first embodiment, and therefore, theirdetailed description is omitted.

Next, the detailed configuration of the image processing section 5 willbe described using FIG. 13. As described above, the image processingsection 5 takes in image data output from the imaging section 2 andperforms various kinds of image processing, such as thinning processingand trimming processing, however, the configuration relating togeneration of image data for the reverse playback of a motion picture inthe art mode is mainly described here.

The temporary recording section 5 a is a memory for temporarilyrecording image data from the imaging section 2 and has a memory 0 fortemporarily recording image data of a still image and a memory 1 to amemory n for temporarily recording image data corresponding to eachframe of a motion picture captured (continuous shooting) following thestill image. The memory 0 for temporarily storing image data of a stillimage is connected to a still image compressing section 5 b. The stillimage compressing section 5 b performs compression of an image, such asJPEG.

The memory 1 to the memory n for recording image data of each frame of amotion picture (continuous shooting) are connected to an order switchingswitch (SW) 5 c. The order switching switch 5 c switches the order ofimage data of each frame temporarily recorded in the memory 1 to thememory n. That is, after a motion picture is recorded, the last imagedata temporarily recorded in the memory n is read first, and then, imagedata is read in order of the memory n-1, the memory n-2, . . . , thememory 2, and the memory 1. Due to this, it is possible to play back animage as if time were reversed.

The order switching switch 5 c is connected to an appropriate imageselecting section 5 d. The appropriate image selecting section 5 dselects images of the image data temporarily recorded in the memory 1 tothe memory n, which are suitable for a motion picture (continuousshooting), for example, images other than those in which the compositionchanges rapidly or blurred images. The appropriate image selectingsection 5 d is connected to an image duplicating section 5 e. As will bedescribed later, when photographing is completed, a motion picturecaptured for 0.5 seconds is expanded to two seconds and a playbackdisplay is produced. The image duplicating section 5 e performsduplication while interpolating image data of each frame of the motionpicture to expand the time of the image.

The image duplicating section 5 e is connected to a motion picturecompressing section 5 g. The motion picture compressing section 5 gperforms motion picture compressing on the image data expanded to twoseconds by the image duplicating section 5 e with MPEG 4, motion JPEGetc. A connection data section 5 f connects the image data of the motionpicture generated by the motion picture compressing section 5 g and theimage data of the still image generated by the still image compressingsection 5 b and generates information about connection for continuouslyplaying back both the image data. The information about connection ofthe connection data section 5 f, the image data from the still imagecompressing section 5 b, and the image data from the motion picturecompressing section 5 g are sent to the recording section 6 and recordedas one image file.

Next, photographing in the art mode by the camera 10 according to thepresent embodiment will be described using FIG. 14. An image 23 a shownin FIG. 14 is an image captured when a photographer determined acomposition, photographing parameters (part of which were automaticallydetermined), and a timing to photograph, and photographed the subjectperson 21 in a still image.

An image 23 b to an image 23 d are images sequentially captured as amotion picture (continuous shooting) following the image 23 a capturedas a still image. The essential photographing is completed in the stillimage, and therefore, the photographing parameters determined by thephotographer or determined automatically by the camera 10, thecomposition determined by the photographer, etc., may be invalidated andthe image in the motion picture is more interesting because there is amotion therein. However, it is not easy to move the subject, andtherefore, in the present embodiment, the photographing parameters arechanged and thus the image is made dynamic.

In the examples shown in the image 23 b to the image 23 d, the effect ofthe change of photographing parameters is given, which is obtained by aclose-up with an overexposure. As the operation to change thephotographing parameters, it may also be possible to shift the focusposition of the photographic lens. By shifting the focus position, suchan effect that the face of a person becomes hazier can be obtained. Inthis case, this may be done along with the change in exposure correctionvalue and zoom, or may be done alone. It may also be possible to blurthe background by changing the aperture.

As described above, in the photographing in the art mode in the presentembodiment, an image at a critical moment is captured in a still imageand a motion picture is captured for a brief time after that. With thephotographing in the art mode, it is possible for a user not havingexperienced photographing of a motion picture to easily enjoy dynamicmotion picture photographing. With the photographing in the art mode, itbecomes possible for a user to capture a varied picture without the needto manipulate the photographing parameters, such as the aperture.

Further, in the present embodiment, as will be described later, it isalso possible to display how an image is formed on the display section 8using image data acquired in the art mode after the photographing. If auser sees an image to which an effect by the art mode is given beforephotographing, it becomes difficult for the user to determine acomposition and photo opportunity.

Next, a playback display produced by combining a plurality of imagescaptured in the art mode (still image with a motion picture) describedin FIG. 14 will be described using FIG. 15. When the still image with amotion picture shown in FIG. 14 is played back and displayedcontinuously as shown in FIG. 15, it is possible to enjoy the image as aslide show like a motion picture with the change in facial expressionetc.

In an example shown in FIG. 15, after the still image photographing(refer to images 24 a, 24 e), zooming is performed while opening theaperture and thus images of the motion picture in which the backgroundis blurred (image 24 b to image 24 d, image 24 f to image 24 h) areobtained. At this time, it is possible to enjoy how the facialexpressions of the persons 21, 22 change.

At this time, it may also be possible to add an effect that seems to beliked by the person photographed instead of simply giving the transitioneffect at all times. For example, it may also be possible to addchanges, for example, the exposure for a person A, the focus positionfor a person B, the zoom for a person C, and the aperture for a personD. It is of course possible to switch the transition effect to anotherby determining when the facial expression changes, or when the personblinks.

The image data used to produce a playback display as shown in FIG. 15may be edited before playback, however, in the present embodiment, insteps S219, S221 (refer to FIG. 19) to be described later, the imagedata is generated as an image file when the image data is recorded atthe time of photographing. When the photographing points of time areclose to each other, both the image data are combined. It may be, ofcourse, possible to enjoy the image data as a still image with thetransition effect in which the image fades out when viewing the image,rather than editing the image data into a still image with a pluralityof motion pictures.

Next, a case is described using FIG. 16, where a still image with amotion picture is handled as a motion picture, an auxiliary image, andreverse playback is performed. In the reverse playback, a moving imageis handled as a decorated image or art image of the face of a personrather than an accurate recorded image. In an example shown in FIG. 16,the motion picture captured in FIG. 14 is played back from the framecaptured last and a still image captured first is displayed as a finalpiece. It may also be possible to represent in a motion picture thestate before a finished image is set as a still image.

That is, when photographing, a still image 26 d is captured first andthen, images 26 c, 26 b and 26 a are captured as a motion picture inthis order for 0.5 seconds. When the captured images are played backreversely, the motion picture is played back reversely for two secondsfrom the image 26 a captured last to the images 26 b and 26 c in thisorder and finally, the image 26 d captured as a still image is displayedas a critical piece.

Recently, an image sensor has been developed, which can capture imagesof hundreds of frames per second, and a playback in such a slow motioncan be performed easily. Whether or not such a reverse playback ispermitted can be determined by determining the change in facialexpression of a subject and referring to the facial expression changeDB6 a, and the reverse playback may be permitted only when the change infacial expression is not unnatural in the reverse playback. For example,in the case of a repetitive motion of, such as blinking, it may also bepossible to arrange a motion picture in the reverse direction before astill image and a motion picture in the forward direction after thestill image, and thus enabling a repetitive playback.

In the art mode in the present embodiment, after a still image iscaptured, a subject is also photographed in a motion picture (continuousshooting), and therefore, if the camera 10 is not directed toward thesubject for a predetermined period of time after the still imagephotographing (0.5 seconds in the embodiment), an unsightly image may beobtained. Hence, in the present embodiment, the person 21 positioned inthe center of the screen is extracted automatically and recorded withthe face part of the person 21 being trimmed, in order not to apply astress more than necessary to a photographer.

A trimming method will be described using FIG. 17 and FIGS. 18A to 18F.FIG. 17 shows the way to photograph the subject person 21. A still imageis captured in a photographing region 15 a and a motion picturefollowing this is captured in photographing regions 15 b and 15 c. Atthis time, among the images captured by the camera 10, FIG. 18A is animage corresponding to the photographing region 15 a, FIG. 18B is animage corresponding to the photographing region 15 b, and FIG. 18C is animage corresponding to the photographing region 15 c.

The image in FIG. 18A corresponding to the photographing region 15 a ofthe still image is recorded as it is as shown in FIG. 18D. As for theimage corresponding to the photographing region 15 b of the motionpicture, however, a trimming region 17 b (the face of the person 21 isin the center) including the screen center of the photographing region15 a is recorded as shown in FIG. 18E. As for the image corresponding tothe photographing region 15 c of the motion picture, a trimming region17 c including the screen center of the photographing region 15 a isrecorded as shown in FIG. 18F. Here, the aspect ratios of the trimmingregions 17 b and 17 c may be different from those of the still images,however, if they are the same, uncomfortable feeling can be eliminated,and therefore, in the present embodiment, the same aspect ratio is used.

As described above, even if a photographer changes the compositions tothe photographing regions 15 a to 15 c, the image data in the trimmingregions 17 b and 17 c corresponding to the center part of the screen ofthe still image is recorded in the recording section 6. Even if aphotographer feels easy after photographing a still image andunconsciously changes the compositions to the photographing regions 15 band 15 c, the center part of the screen is trimmed and recorded. Bytrimming and using only the face part, it is possible to represent amotion picture for capturing the change in facial expression of thesubject. That is, as shown in FIGS. 18D to 18F, it is possible to recordthe history in which the subject comes to have a relaxed facialexpression in the transition of images and to recall the memory.

Next, the operation of the camera 10 according to the present embodimentwill be described using the flowchart of camera control shown in FIG.19. When the camera 10 is powered on and started, the flow shown in FIG.19 starts. When the flow starts, first, whether or not the mode is thephotographing mode is determined (S201). The camera 10 comprises thephotographing mode and the playback mode, and the mode state selected bya photographer is determined based on the operation state of theoperation section 9.

When the mode is the photographing mode according to the result of thedetermination in step S201, an image is captured and displayed (S202).In this step, the image data acquired by the imaging section 2 issubjected to image processing in the image processing section 5 and alive view is displayed on the display section 8. It is possible for aphotographer to determine a composition and photo opportunity whileviewing the live view display.

After the image is captured and displayed, next, the screen features aredetermined (S203). In the screen feature determination, the facedetecting section 2 a detects whether or not the face part is includedin the image and when included, detects the position and size thereof.When the face part is detected, the exposure control values, such as theaperture value and the shutter speed, are calculated so that the partachieves an appropriate exposure, and the automatic focus adjustment ismade to focus the part.

After the screen feature determination is made, next, whether or notphotographing is performed is determined (S204). Here, whether or notthe release button of the operation section 9 is pressed and operated isdetermined. When photographing is to be performed according to theresult of the determination, still image photographing is performed(S205). In this step, the image data output from the imaging section 2is temporarily recorded in the memory 0 within the temporary recordingsection 5 a of the image processing section 5 and at the same time, theimage is compressed in the still image compressing section 5 b.

After the still image photographing is performed, next, whether or notthe mode is the art mode is determined (S206). The art mode is set byoperating the art mode button of the operation section 9, and therefore,in this step, whether or not the art mode is set by operating the artmode button is determined. When the mode is not the art mode accordingto the result of the determination, the still image is recorded (S207).In this step, the image data of the still image acquired in step S205and subjected to image compression processing is recorded in therecording section 6. After the still image is recorded, the processreturns to step S201.

When the art mode is set according to the result of the determination instep S206, next, continuous photographing (motion picture) is started(S211). Here, continuous photographing (motion picture) to repeatedlyacquire the image data from the imaging section 2 at a predeterminedtiming is started.

Subsequently, an exposure switch direction step determination is made(S212). When the art parameters are changed, to be described later, thephotographing parameters are switched while performing continuousphotographing, and at this time, it is necessary to change the apertureas a photographing parameter. In FIG. 14 described above, the example isshown, in which the exposure is shifted toward the overexposure side,however, in a dark scene, there is no room to shift the exposure towardthe overexposure side, and in this case, the underexposure side ispreferable. In addition to the effect in which an image is overexposedand disappears in light, such an effect may be recommended in which animage is underexposed and disappears in the dark, and emerges againtherefrom. Hence, in the exposure switch direction step determination,the direction in which there is a room for exposure is determined andtoward which of the overexposure side and the underexposure side theexposure is shifted is determined. Details of the exposure switchdirection step determination will be described later using FIG. 22. Whenthe photographing parameters are changed, it may also be possible toswitch in accordance with the facial expression of the subject.

After the exposure switch direction step determination is made, next,whether or not 0.5 seconds have elapsed is determined (S213). Here, thetime from starting the continuous photographing in step S211 is measuredby the clock section 11 and thus whether or not 0.5 seconds have elapsedis determined.

When 0.5 seconds have not elapsed according to the result of thedetermination in step S213, the art parameters are changed (S214). Here,the photographing parameters are changed while performing continuousphotographing. In the present embodiment, a live view is displayed alsoduring the continuous photographing, and therefore, it is possible toconfirm the change in captured images accompanying the change of thephotographing parameters. However, the change occurs in a brief time,and the change is too fast for the human eyes to notice, and therefore,0.5 seconds are expanded to two seconds and the change is displayed likea slow motion. Details of the change of the art parameters will bedescribed later using FIG. 20. While changing the art parameters, let0.5 seconds elapse.

When 0.5 seconds have elapsed according to the result of thedetermination in step S213, next, the continuous photographing is endedand a continuing display is provided (S215). Because 0.5 seconds haveelapsed, the continuous photographing is ended, however, the time isexpanded to two seconds in the change of the art parameters in step S214and the (motion picture of) continuous shooting in the art mode isplayed back and displayed, and therefore, this playback display iscontinued.

Subsequently, whether or not there is a motion during the continuousshooting is determined (S216). Whether or not there is such a motion asdescribed in FIG. 17 during the time from starting the continuousphotographing in step S211 to ending the continuous photographing instep S215 is determined based on the motion vector in the imageprocessing section 5.

When there is a motion during the continuous shooting according to theresult of the determination in step S216, the image is trimmed so thatthe part located in the center of the screen is centered (S217). Here,as described in FIG. 17, the center of the screen of the still image istrimmed by the trimming section 1 c. It may also be possible to trim theimage so that the face part detected by the face detecting section 2 ais located in the center instead of the center of the screen.

When the image is trimmed in step S217, or when there is not a motionduring the continuous shooting according to the result of thedetermination in step S216, next, the images continuously shot areedited into a motion picture in the reverse direction while performinginterpolation (S218). Here, as described using FIG. 16, the order of theimage data of each frame shot by continuous shooting is reversed in theimage processing section 5 and the order of the frames is sorted fromthe frame shot last in a time-reversed order. At this time, aninterpolated image is inserted between each frame and the photographingtime of 0.5 seconds is expanded to two seconds.

Subsequently, the motion picture and the still image are recorded(S219). Here, data used to combine the motion picture (continuousshooting, images captured continuously) and the still image is attachedby the connection data section 5 f (refer to FIG. 13) and the stillimage and the motion picture as an auxiliary image are recorded in therecording section 6. At this time, information about photographing inthe art mode is also attached.

After the motion picture and the still image are recorded, next, whetheror not the time having elapsed from the previous recording is within apredetermined time is determined (S220). Here, whether or not the timefrom completing the previous continuous photographing (S215) to startingthe next continuous photographing (S211) is within a predetermined timeis determined. When the time is not within a predetermined timeaccording to the result of the determination, the process returns tostep S201.

When the time is within a predetermined time according to the result ofthe determination in step S220, next, connection with the previousmotion picture is made (S221). As described using FIG. 15, inpreparation for a case where a still image to which a motion picture isattached as an auxiliary image is played back continuously, one fileincluding them is generated in this step. Regarding a predeterminedtime, when a case is considered, where a motion picture includingseveral images in one hour is created, it may also be possible toconnect still images captured within ten minutes.

It may also be possible to change the way of connection in accordancewith the transition effect. For example, if a pure white overexposedscreen is connected after a deep-black screen, it is not natural, andtherefore, after the screen is transitioned to a deep-black one, it mayalso be possible to display a still image and then connect a motionpicture in the forward direction toward overexposures rather thandisplaying the transition motion picture in the reverse direction fromthe overexposure side. After the connection with the previous motionpicture is made in step S221, the process returns to step S201.

When the mode is not the photographing mode according to the result ofthe determination in step S201, next, whether or not the mode is theplayback mode is determined (S231). The playback mode is set by pressingand operating the playback button of the operation section 9, andtherefore, in this step, whether or not the playback mode is set isdetermined. When the mode is not the playback mode according to theresult of the determination, the process returns to step S201.

When the mode is the playback mode according to the result of thedetermination in step S231, next, whether or not the mode is the artmode is determined (S232). In the recording section 6, not only theimage data photographed by the normal still image photographing but alsothe image data photographed in the art mode is recorded. In this step,whether or not the currently selected image is captured in the art modeis determined based on the information attached to the image data.

When the mode is the art mode according to the result of thedetermination in step S232, the corresponding motion picture part isplayed back in a slow motion (S233). In the playback in the art mode, asdescribed using FIG. 16, reverse playback is performed on the displaysection 8 in a time-reversed order sequentially from the motion picturecaptured last. At this time, the playback time is lengthened longer thanthe motion picture photographing time and the motion picture is playedback like a slow motion.

When the corresponding motion picture part is played back in a slowmotion in step S233, or when the mode is not the art mode according tothe result of the determination in step S232, next, a still image isplayed back (S234). In the case of a normal still image, the imageselected in this step is enlarged to be played back and displayed on thedisplay section 8. When the mode is the art mode, the auxiliary motionpicture is played back in step S233 (corresponding to the images 26 a to26 c in FIG. 16) and in this step S234, the still image (correspondingto the image 26 d) is played back as a finished image. When the stillimage is played back, the process returns to step S201.

As described above, in the example in the present embodiment, whenphotographing, immediately after the still image photographing,continuous shooting (motion picture photographing) is performedsuccessively for five seconds as shown in FIG. 23A, and in playback(live view), the continuously shot part (transition motion picture) islengthened to two seconds and displayed as shown in FIG. 23B. Inplayback, the same image of the transition motion picture is displayedseveral times and a display like a slow motion is produced. This isbecause, in consideration of camera shake of a photographer, thecontinuous shooting is desirably about 0.5 seconds, however, if it isplayed back as it is, it is difficult to recognize the image because ofits high speed.

Next, the subroutine of changing art parameters in step S214 will bedescribed using the flowchart shown in FIG. 20. When the flow ofchanging art parameters is started, first, photographing is performed(S241). In step S211, continuous photographing is started and in thisstep, an image of one frame constituting a continuous shot image (motionpicture) is captured and the image data obtained at this time istemporarily recorded in any of the memory 1 to memory n of the temporaryrecording section 5 a sequentially.

Subsequently, a motion is detected and whether or not there is a motionis determined (S242, S243). Here, the motion determining section 4determines whether or not there is camera shake or a change incomposition, however, this is not limited and it may also be possible tomake the determination based on a motion vector detected by the imageprocessing section 5.

When there is a motion according to the result of the determination instep S243, next, whether or not a still image and an image are differentis determined (S251). Here, the determination is made by comparing thestill image captured by the still image photographing in step S205 withthe image captured by the motion picture photographing in step S241.This is because if the difference between the still image and the motionpicture is large, the continuity is impeded and it is not worth whileviewing as a series of images. When the image is different from thestill image according to the result of the determination in step S251,the process advances to step S215 and the continuous photographing isended.

When there is not a motion according to the result of the determinationin step S243, next, whether or not there is a room to change theaperture is determined (S244). Here, whether or not there is a room toopen the aperture toward the fully open side from the currently setaperture value is determined.

When there is a room to change the aperture according to the result ofthe determination in step S244, next, the aperture is opened in such away that the aperture is opened by one step for five frames (S245). Theexposure is appropriate when a still image is captured, and however,when the aperture is opened gradually, the image gradually becomes anoverexposed one. In the present embodiment, the aperture is opened byone step for five frames, however, this is an example, and the way maybe changed appropriately. In the present embodiment, photographing of 30frames per second is supposed, however, in the case of photographing of60 frames per second, it may also be possible to open the aperture byone step for ten frames, and it may also be possible to appropriatelyset the number of frames for which the aperture is opened by one step inaccordance with other conditions.

When there is not a room to change the aperture according to the resultof the determination in step S244, whether or not there is a room tochange the zoom is determined (S252). Here, the information about thecurrent focal length of the zoom lens is acquired and then whether ornot the focal length can be changed toward the longer focal length sideis determined.

When there is a room to change the zoom according to the result of thedetermination in step S252, next, the focal length is sequentiallyincreased by a factor of 1.1 for each frame (S253). The parametercontrol section 7 is instructed to increase the focal length by a factorof 1.1 each time it performs step S253. The factor of 1.1 is an example,and when, for example, high-speed continuous shooting is available, thefocal length may be increased by a factor of 1.1 for two frames. Thefactor may be appropriately changed in consideration of the zoomingspeed, continuous shooting speed, etc.

When there is not a room to change the zoom according to the result ofthe determination in step S252, or when the still image and the imageare not different according to the result of the determination in stepS251, next, an exposure switch step is determined from the remainingtime (S254). In this case, the exposure correction value is changed as aphotographing parameter. A detailed flow of the exposure switch stepdetermination will be described later using FIG. 21.

Subsequently, the exposure is switched to another sequentially inaccordance with the exposure correction value determined in step S254(S255). Here, the aperture, the shutter speed, etc., are changed usingan exposure correction value ΔE calculated in step S254.

When the aperture is opened sequentially in step S245, or when zoomingin is performed sequentially in step S253, or when the exposure isswitched to another sequentially in step S255, next, the images capturedby photographing are displayed shifted from one another (S246). Asdescribed above, the continuous photographing (motion picture) isperformed for 0.5 seconds, however, the time is too brief to recognizethe transition effect by changing the photographing parameter, andtherefore, a display is produced in a slow motion so that the displaytime is about two seconds. After the images captured by photographingare shifted, the process returns to the original flow.

Next, the subroutine of the exposure switch step determination in stepS254 is described using the flowchart shown in FIG. 21. When the flow ofthe exposure switch step determination is started, first, time ismeasured and at the same time, a remaining time t of a motion picture iscalculated (S261). When continuous photographing is started in stepS211, steps S213 and S214 are repeated for 0.5 seconds and in themeantime, image data of frames constituting the motion picture isacquired in step S241. Hence, in step S261, the remaining time t of themotion picture is found by subtracting the time having elapsed from thestart of continuous photographing in step S211 from 0.5 seconds.

Subsequently, a difference E from the exposure of the still image isfound (S262). Here, the difference E in the amount of exposure shiftedfrom the amount of exposure at the time of the still image photographingin step S205 is calculated. Each time the subroutine of the exposureswitch step determination is executed, the exposure is corrected andreduced by the amount of exposure E from that of the still image, andtherefore, the reduced amount of exposure E is found here.

After the difference E from the exposure of the still image is found,next, an amount of change in exposure ΔE per unit time is found (S263).The amount of change in exposure ΔE is calculated by (three steps−E)/t.Here, the three steps is an aperture value, E is the difference in theamount of exposure found in step S262, and t is the remaining time ofmotion picture photographing found in step S261. In the presentembodiment, the camera is designed so that when motion picturephotographing is ended, the exposure correction by three steps isperformed for the still image photographing and in step S263, the amountof change in exposure ΔE per unit time is found with which the exposurecorrection by three steps is performed when motion picture photographingis ended. Three steps is an example, and different values may beaccepted in accordance with various conditions, such as the facialexpression.

After the amount of change in exposure ΔE is found, next, an amount ofexposure correction for each frame is found (S264). Here, it is found bydividing the amount of change in exposure ΔE for each frame by thenumber of frames in a unit time. After this calculation, the processreturns to the original flow and the exposure is corrected in accordancewith the amount of exposure correction ΔE (exposure correction value)for one frame.

Next, the subroutine of the exposure switch direction step determinationin step S212 is described using the flowchart shown in FIG. 22. When theflow of the exposure switch direction step determination is started,first, a direction toward which there is a room to change exposure isdetermined (S271). Here, whether the exposure can be transitioned towardthe overexposure side or toward the underexposure side when the exposureis corrected is determined. In the case of a night scene, the scene isdark originally, and therefore, it is difficult to overexpose the sceneto a pure white one even by transitioning the exposure toward theoverexposure side because of the exposure limit or sensitivity limit ofone frame. Rather than that, the transition in which the darknessdeepens is more effective and natural. Hence, toward which side there isa room to change the exposure is determined based on the amount ofexposure, which is an appropriate exposure at the time of still imagephotographing.

After the direction toward which there is a room to change the exposureis determined, next, whether there is a room to change exposure by threesteps toward the overexposure side is determined (S272). Here, thedetermination is made based on the size of the room in step S271. Whenthere is a room to change exposure by three steps toward theoverexposure side according to the result of the determination, theexposure is transitioned toward the overexposure side (S273). On theother hand, when there is not a room to change exposure by three stepstoward the overexposure side, next, the exposure is transitioned towardthe underexposure side (S274).

When the transition toward the overexposure side is made in step S273 orwhen the transition toward the underexposure side is made in S274, theprocess returns to the original flow. When the transition toward theoverexposure side is made, the amount of exposure correction is shiftedtoward the overexposure side and when the transition toward theunderexposure side is made, the amount of exposure correction is shiftedtoward the underexposure side.

As described above, in the fourth embodiment of the present invention,when the art mode is set, image data is acquired as a still image andfollowing this, while the photographing parameters are changed to onesdifferent from those at the time of the photographing of the stillimage, image data is acquired by continuous shooting. Then, the imagedata of the still image and the continuous shooting acquired here isdisplayed in a representation different from the time flow with whichimages are captured by the continuous shooting at a timing in connectionwith the still image. Hence, it is possible to fuse the still image andthe motion picture to capture and play back a varied image.

Further, in the fourth embodiment of the present invention, it ispossible to capture an image to which the transition effect is attached,and therefore, it is possible to easily create a motion picture thatcombines a plurality of cuts that includes no rapid change in scene.Further, it is also possible to record a favorite image, which cannot berepresented by a still image, as one image to which the effect of changeis attached by changing the angle of view etc.

In the fourth embodiment of the present invention, as the photographingparameters, the aperture value, the focal length (zoom), and theexposure correction value are described, however, these are not limited,and for example, it may also be possible to sequentially switch thefocus positions. When switching the focus positions at the time of stillimage photographing, for example, it may also be possible to graduallyswitch from the focus position where the main subject is in focus to theposition where the background is in focus, or when there is anotherperson other than the main subject, to the position where the otherperson is in focus. In particular, at the time of still imagephotographing, there is uncomfortable feeling if the subject is not infocus and it is also difficult to determine framing or a timing tophotograph, and therefore, such specifications are reasonable that thephotographing parameters are changed after photographing, as describedin the fourth embodiment.

If the motion picture captured while changing the photographingparameters is played back in the reverse direction as shown in FIG. 16,a dramatic effect that the subject is becoming in focus can be obtained.The direction in which the focus is adjusted is determined by the focusposition of the subject and the limit of focus adjustment of thephotographic lens, and when the subject is very near, the focus isadjusted toward the distant side and when snap photographing isperformed, the focus is adjusted to the near side, and thus, the amountof blurring becomes large and a vivid effect can be obtained. It may ofcourse be possible to disable such specifications because of theoccurrence of a time lag at the time of continuous shooting.

In the fourth embodiment of the present invention, as the photographingparameters, the aperture value, the focal length (zoom), and theexposure correction value are described, however, it is acceptable iftwo of them or only one of them can be performed when all of the threecannot be performed. In the fourth embodiment of the present invention,the degree of blurring as a photographing parameter is changed bycontrolling the aperture value, however, it may also be possible to makeuse of image processing when blurring the background through imageprocessing.

Further, in the fourth embodiment of the present invention, by simplychanging the aperture and exposure correction value as the photographingparameters, an overexposure is achieved, however, this is not limitedand it may also be possible to change the drive direction of thephotographing parameters in accordance with a facial expression. Forexample, it may also be possible to underexpose when the facialexpression becomes gloomy, or to increase the zooming speed when thefacial expression becomes cheerful.

Furthermore, in the fourth embodiment of the present invention,continuous shooting (motion picture) is performed for 0.5 seconds and itis expanded to two seconds when played back and displayed, however,these values are examples, and other times may be accepted, of course.However, it is desirable for the time of continuous shooting (motionpicture) to be such one which is not affected by any influence of camerashake. Other set values are examples, respectively, and values otherthan those set values may be accepted.

Still furthermore, in the fourth embodiment of the present invention,the image captured in the art mode is displayed on the display section 8of the camera 10 immediately after the photographing or at the time ofplayback. The playback display may be produced, however, by a personalcomputer, for example, rather than the camera 10. In this case, theimage data recorded in the recording section 6 may be played back anddisplayed as it is, and it may also be possible for the camera 10 not toperform image processing for reverse direction motion picture playbackbut to cause a personal computer to perform image processing forreversing the motion picture.

Next, a fifth embodiment of the present invention will be describedusing FIG. 24 to FIG. 29. When starting photographing, a photographingdevice in the fifth embodiment captures a motion picture whilesequentially focusing each subject in an image to be captured and endsphotographing after photographing a still image finally. A still imageis captured last after a motion picture is captured, and therefore, avaried image can be obtained.

FIG. 24 is a block diagram showing an electric circuit of the cameral 10according to the fifth embodiment of the present invention. The camera10 is a digital camera and includes the control section 1, the imagingsection 2, the face detecting section 2 a, the parameter setting section3, the motion determining section 4, the recording section 6, theparameter control section 7, the display section 8, the operationsection 9, the flash ROM 12, etc.

Like the imaging section 2 in the first embodiment, the imaging section2 includes a photographic lens (zoom lens) having a zoom function, anexposure control section, such as a shutter and aperture, an imagesensor, a drive circuit and a read circuit of an image sensor, etc., andconverts a subject image formed by the photographic lens into image databy the image sensor and outputs the data. Further, the photographic lensof the imaging section 2 includes a focus adjustment lens and itsmechanism and in the present embodiment, it is also possible tophotograph while shifting a focus by the focus control section 7 b, tobe described later, when photographing a transition motion picture.Furthermore, an aperture adjustment mechanism for blurring thebackground may be provided. The face detecting section 2 a is the sameas that in the first embodiment, and therefore, its detailed descriptionis omitted.

The control section 1 controls the total sequence of the camera 10 inaccordance with programs stored in the flash ROM 12. The control section1 has the facial expression change extracting section 1 a, thecomposition change determining section 1 b, a contrast change extractingsection 1 d, a region dividing section 1 e, and the image processingsection 5. The facial expression change extracting section 1 a is thesame as that in the first embodiment, however, it may also be possibleto cause the facial expression change extracting section 1 a to predictthe change in facial expression based on the speed of the change whenthe facial expression changes when determining the change in facialexpression. Further, the composition change determining section 1 b isthe same as that in the first embodiment, and therefore, its detaileddescription is omitted.

The contrast change extracting section 1 d extracts a high-frequencycomponent (contrast) from image data output from the imaging section 2and extracts the change in contrast. As will be described later, it ispossible to obtain information about distance to subject by the contrastchange extracting section 1 d detecting a peak value of the change incontrast when scanning the photographic lens.

The region dividing section 1 e divides the region of the screen basedon the information about distance to subject obtained by theabove-mentioned contrast change extracting section 1 d. That is, theimage data output from the imaging section 2 is divided into the form ofa mesh and information about distance to subject is obtained at eachpoint, and thus, it is possible to divide the screen region into eachpoint having substantially the same distance to subject. The regiondivision by the region dividing section 1 e will be described laterusing FIGS. 27A to 27C.

The image processing section 5 in the control section 1 has the motionpicture section 5 h and the still image section 5 i, takes in image dataoutput from the imaging section 2, performs various kinds of imageprocessing, such as thinning processing, trimming processing, edgeenhancement, color correction, and image compression, and performs imageprocessing for displaying a live view of a motion picture, for recordinga still image and a motion picture in the recording section 6, forplaying back and displaying a still image and a motion picture, etc. Forcompression/decompression of an image etc., the image processing section5 has circuits for a still image and a motion picture in accordance witheach processing, such as a still image compressing/decompressing circuitand a motion picture compressing/decompressing circuit. Further, it ispossible for the image processing section 5 to acquire color informationand a motion vector based on image data.

The parameter control section 7 has the focus control section 7 b. Thefocus control section 7 b adjusts the focus position of the photographiclens of the imaging section 2 so that the contrast information detectedby the contrast change extracting section 1 d is indicative of the peakposition. That is, a so-called automatic focus adjustment by contrastmethod is made. In the art motion picture mode, as will be describedlater, the focus adjustment of the photographic lens is made so that thefocus moves from the secondary subject to the main subject.

The operation section 9 includes various operation members, such as therelease button, the power switch, the playback mode setting button, andthe menu button, and determines the operation state of these operationmembers and sends the determination result to the control section 1. Thecontrol section 1 described above controls photographing and playback ina predetermined sequence in accordance with the operation state of theoperation member.

Like the display section 8 in the first embodiment, the display section8 has a display section, such as a liquid crystal panel and an organicEL panel, disposed on the backside etc. of the camera 10 and produces adisplay of a live view before photographing, a display of normalplayback of a recorded image, a display of camera information, etc. Itis possible for a photographer to determine a composition and timing byobserving the live view display. Further, the display section 8 in thepresent embodiment also plays back and displays a transition imageobtained in the art motion picture mode, that is, by capturing a motionpicture while moving the focus and finally photographing a still image.

The parameter setting section 3, the parameter control section 7, themotion determining section 4, the recording section 6, the facialexpression change DB 6 a, and the flash ROM 12 are the same as theparameter setting section etc. in the first embodiment, and therefore,their detailed description is omitted.

Next, the photographing in the art motion picture mode by the camera 10according to the present embodiment is described using FIG. 25A, FIG.25B and FIG. 26. The art motion picture mode is a mode in which adistance to a subject etc. in the screen is measured when photographingand if there is another distance to a different subject, a main subjectand a secondary subject are selected and the focus is moved from theselected secondary subject to the selected main subject, and a motionpicture is captured during the movement of the focus and finally themain subject is photographed in a still image.

As shown in FIG. 26, it is assumed that a person 21 a, as a mainsubject, is substantially in the center of the screen and there is aflower 21 c on the near distance side and a tree 21 b, as a secondarysubject, in the background (on the long distance side). Whenphotographing is performed in the art motion picture mode in such alayout, an image 25 a is obtained first, in which the tree 21 b in thebackground is in focus as shown in FIG. 25A and then, the focus positionmoves gradually toward the person 21 a and in the meantime, image datais recorded in a motion picture. Then, an image 25 b in which the person21 a is in focus as shown in FIG. 25B is obtained finally, and imagedata is recorded as a still image. This is an effective method toindicate what is present in the background in a pretty image (refer toFIG. 25B) in which the background is blurred. It is desirable to focusthe main subject (the person 21 a in the example in FIG. 25B) in thefinal image and this image is recorded as a still image.

As described above, in the art motion picture mode in the presentembodiment, image transition is performed, and in the transition fromFIG. 25A to FIG. 25B, the image is changed smoothly and as the image 25a in FIG. 25A and the image 25 b in FIG. 25B, different images thedifference between which can be recognized at a glance are selected. Inthe image transition, if the focus position is moved back and forthrepeatedly, a viewer will be displeased, and therefore, such a movementof the focus should be avoided, in which, for example, the focus ismoved from the flower 21 c to the person 21 a, then the tree 21 b isfocused, and then the person 21 a is focused again.

Consequently, it is necessary to select a secondary subject and a mainsubject as a start point and an end point of the image transition.Either of the main subject and the secondary subject may be selected asthe start point, however, in the present embodiment, the transitionstarts from the secondary subject and ends at the main subject because acompleted still image is to be obtained finally. As a method ofselecting a main subject and a secondary subject, there can be conceivedof various methods, and in the present embodiment, a combination thatexhibits the effect of focusing remarkably is preferred.

Next, the selection of the main subject and the secondary subject isdescribed using FIGS. 27A to 27C. FIG. 27A is a graph showing arelationship between distance to subject and contrast, FIG. 27B shows animage of the subject, and FIG. 27C shows divided image regions. Asdescribed above, in the present embodiment, information about distanceto subject is acquired from the peak position of the contrast when thephotographic lens is scanned. The photographic lens can be scanned fromthe nearest end to the infinite end, however, it takes time to scan theentire region, and therefore, in the present embodiment, only part ofthe region is scanned and other regions are scanned at another timing orinferred. It may also be possible to make use of information aboutdistance of focusing by the contrast method when displaying a live view.

In the example shown in FIGS. 27A to 27C, for the focus adjustment ofthe photographic lens, the photographic lens is scanned at a pluralityof points 22 a, 22 b and 22 c in the screen as shown in FIG. 27B andthus the contrast is measured. The changes in contrast at each of thepoints 22 a, 22 b and 22 c are represented by curves as shown in FIG.27A. As described above, the photographic lens is not scanned over theentire region but in the range of focus positions Z1, Z2 and Z3 in theexample shown in FIG. 27A.

As can be seen from FIG. 27A, the contrast curve 22 a corresponding tothe person 21 a has its peak at the focus position Z2, the contrastcurve 22 b corresponding to the flower 21 b has its peak on the moredistant side of the focus position Z2, and the contrast curve 22 ccorresponding to the tree 21 c has its peak on the nearer side of thefocus position Z2.

In the example shown in FIGS. 27A and 27B, the contrast is measured atonly the three points for the convenience of explanation, however, ifthe contrast at more points of a more finer mesh is acquired and thechange in contrast and information about contour are obtained, it ispossible to divide the screen into individual regions in accordance witha subject as shown in FIG. 27C. In the example shown in FIG. 27C, thescreen is roughly divided into a region 23 a in which the person 21 a ispresent, a region 23 c before the region 23 a, in which the flower 21 cis located, and a background region 23 b in which the tree 21 b isincluded.

When the screen is divided into each region, various pieces ofinformation can be obtained for each region, such as whether a face isincluded, which region is wide, color components included in eachregion, and whether or not there is motion. By using these pieces ofinformation, it is possible to determine which region is an importantregion, and therefore, to determine regions that include the mainsubject and the secondary subject described above. Further, whenmeasuring the contrast while moving the focus position of thephotographic lens after measuring the contrast between the focuspositions Z1 to Z3 and dividing the screen into each region, if thecontrast changes rapidly between the focus position Z3 and a focusposition Z4, it is possible to modify the region and make use of theregion in the determination as to whether the region is an importantregion by referring to an amount of change in contrast h and a width ofchange from the focus position Z2 where the face of the person 21 a isfocused at this time.

When there is a subject with high contrast (in the example in FIG. 27B,the tree 21 b at the focus position Z4) at a position different from thefocus position of the face (in the example in FIG. 27B, the focusposition Z2), it is effective when the effect of focus shift isrepresented in a motion picture, and therefore, in the presentembodiment, information about contrast in the position where the focuschanges is attached importance. This is because when the change in focusis small, the effect of focus shift is small and there is a possibilitythat a viewer does not notice it.

Selection of a main subject and a secondary subject is determined basedon weight values by weighting the divided regions, respectively,described in FIG. 27C. FIG. 28 shows weight values. When assigningweights, first, Za for determining a main subject is weighted. Whenweighting Za, whether or not the conditions shown in FIG. 28 aresatisfied is determined for each of the regions 23 a to 23 c.

For example, as to the first condition that “face with the upper half ofthe body”, whether or not the region 23 a satisfies the condition isdetermined. In the example shown in FIG. 27B, the person 21 a in theregion 23 a has the face with the upper half of the body, and therefore,a weight of 10 points is assigned. On the other hand, in the regions 23b and 23 c, there is no face with the upper half of the body, andtherefore, a weight of zero point is assigned. Whether or not there is aface is determined by the face detecting section 2 a and whether or notthe face is one with the upper half of the body is determined by theimage processing section 5. It may also be possible to change the degreeof priority based on not only the size of a face but also the facialexpression of the face. A face with an unnaturally large size, a face inprofile, etc., are given a low degree of priority, however, noexplanation is given here for the sake of simplification of theexplanation.

The third weighting condition that whether or not “there is a motion” inFIG. 28 is determined by the image processing section 5 determining amotion vector. A subject in motion attracts attention of a viewer, andtherefore, a weight of five points is assigned. The fourth weightingcondition is that whether or not “there is a change in color” and thisis determined by the image processing section 5 determining whether ornot a number of colors are included in the region. This is because whena number of colors are included, the region attracts attention of aviewer.

The fifth and sixth weighting conditions relate to the size of theregion area and are determined by the image processing section 5. Whenthe area is large, it is considered that the region includes animportant subject. The eighth weighting condition is that whether or not“screen center part” and this is determined by the image processingsection 5. When a subject is in the screen center, this is doneintentionally by a photographer in most cases and it can be consideredthat the subject is important.

As described above, the conditions are determined for each of theregions 23 a to 23 c and weight is assigned based on the determinationresult, the total of the points assigned is calculated, and then theregion with the largest total value is determined to be the region Za inwhich the main subject is present. In the example shown in FIG. 27B, theregion 23 a in which the person 21 a is included is the region Za. Whenselecting the region Za, the conditions shown in FIG. 28 are examples,and part of the conditions shown in FIG. 28 may be employed or otherconditions may be added thereto.

When the region Za where the main subject is present is determined,next, the region Zb where the secondary subject is present isdetermined. When selecting the secondary subject, the values in eitherof the columns as shown in FIG. 28 are used depending on whether or nota face is included in the image. Consequently, first, whether or not aface is included in the image is determined and a table to be used isdetermined. Subsequently, as to the regions other than the region wherethe main subject is included, whether or not each condition is satisfiedis determined as in the case of the main subject and when a certaincondition is satisfied, the point described in FIG. 28 is assigned andthe region with the largest total value is determined to be the regionZb where the secondary subject is present.

As the conditions used in determining the region Zb where the secondarysubject is present, the following conditions are added to the conditionsfor the main subject. That is, as the seventh weighting condition,whether or not “brighter than Za part” is determined. When the region Zbis brighter than the region Za of the main subject, this attractsattention of a viewer and the degree of importance can be thought high.Further, as the ninth weighting condition, whether or not “contrast ishigher than face part” is determined. When the contrast is higher thanthe contrast of the face part in the image, there is a possibility thatthe degree of importance is high.

Furthermore, as the tenth weighting condition, whether or not “focus isdistant from focus position” is determined. This is the case where thefocus position is more than a predetermined distance apart from the mainsubject and in such a case, the effect of the focus shift turns outremarkably. This is important in selecting the secondary subject, andtherefore, as high a point as seven is set in the present embodiment. Inselecting the region Zb, the conditions shown in FIG. 28 are examples,and part of the conditions shown in FIG. 28 may be employed or otherconditions may be added thereto.

Next, the operation of the camera 10 in the present embodiment isdescribed using the flowchart of the art motion picture mode shown inFIG. 29. The camera 10 can operate in various modes, however, forsimplicity, in this flowchart, only the art motion picture mode isdescribed and other modes are omitted. For the setting of the art motionpicture mode, it may also be possible to provide a dedicated operationbutton for setting the art operation mode, or use a menu screen forsetting. Further, it may also be possible to design the camera 10 sothat the art operation mode is set as a default value, or the camera 10may be one exclusive for the art operation mode.

The flowchart shown in FIG. 29 is executed by the control section 1 inaccordance with the programs stored in the camera 10. When the flow ofthe art operation mode is started, first, whether or not the releasebutton is pressed halfway is determined (S301). Here, whether or not therelease button of the operation section 9 is in the halfway-pressedstate is determined. If not, let time elapse until the halfway-pressedstate is brought about while displaying a live view on the displaysection 8.

When the release button is in the halfway-pressed state according to theresult of the determination in step S301, the focus position is shifted(S302). Here, for example, the photographic lens is scanned between Z1and Z3 in FIG. 27. When the power switch is turned on, the camera 10starts to display a live view and in the meantime, the focus is adjustedand the contrast of a subject image is monitored real time, and when thecontrast changes, the focus is adjusted again. When the focus positionis shifted in step S302, the range of Z1 to Z3 is determined usinginformation about focus adjustment made when a live view is displayed.

Subsequently, an image is captured (S303). In step S302, the scanning ofthe photographic lens is started and while scanning, the contrast isacquired by the contrast change extracting section 1 d using the imagedata from the imaging section 2 and information about distance tosubject at each point is acquired.

Next, whether or not the measurement of a plurality of focuses iscompleted is determined (S304). Here, as described using FIG. 27, thecontrast is acquired for all of the points in the screen and whether thecollection of information about distance to subject is completed isdetermined. When the collection is not completed according to the resultof the determination, the process returns to step S302.

When the measurement of the plurality of focuses is completed accordingto the result of the determination in step S304, next, the screen isdivided (S305). Here, as described in FIG. 27C, the region of the entirescreen is divided in accordance with the subject based on theinformation about distance to subject etc.

When the region is divided, next, weighting of the region Za isperformed (S306). Here, as described in FIG. 28, in order to select amain subject, the determination of region weighting is made for eachregion divided in step S305 and the region with the largest total valueis selected as the Za region.

When the weighting of the region Za is performed, next, weighting of theregion Zb is performed (S307). Here, as described in FIG. 28, in orderto select a secondary subject, the determination of region weighting isperformed for each region divided in step S305 and the region with thelargest total value is selected as the Zb region.

When the weighting of the region Zb is performed, next, a difference inexposure dE between the Za region and the Zb region is determined(S308). Here, as to the Za region acquired in step S306 and the Zbregion acquired in step S307, the luminance of the region is acquiredusing image data of each region and the difference in appropriate amountof exposure dE is calculated using the luminance.

When the difference in exposure dE is found, next, whether or not therelease button is pressed fully is determined (S309). Here, whether ornot the release button of the operation section 9 is in thefully-pressed state is determined. Let time elapse until thefully-pressed state is brought about and when the fully-pressed state isbrought about, next, the focus and exposure are adjusted to the Zbregion and photographing is performed (S311). The Zb region is theregion of the secondary subject, and first, the focus and exposure areadjusted to this region. The state in this step S311 is the state of theimage 25 a illustrated in FIG. 25A and in this case, the state is suchone where the focus and exposure are adjusted to the tree 21 b in thebackground.

Subsequently, a difference in focus dZ between the Za region and the Zbregion is determined (S312). Here, the difference in focus dZ is found,which corresponds to a difference between information about distance tothe main subject and information about distance to the secondarysubject. The difference in focus dZ is used in step S316.

When the difference in focus dZ is determined, next, whether or notthere is a motion in the Zb region is determined (S313). Here, whetherthere is something that moves in the region where the secondary subjectis present is determined based on the motion vector of the image data.

When there is not a motion according to the result of the determinationin step S313, let time elapse (S314). On the other hand, when there is amotion, let three seconds elapse. When there is a motion, a viewerdesires to confirm what is the moving subject in many cases and hence,the waiting time is set long. One second and three seconds are examples,and the waiting time may be changed appropriately in consideration ofthe preferences of a photographer and a viewer.

After letting a predetermined time elapse in step S314 or S315, next,the focus shift is started toward the Za region and in this case, aspeed in accordance with the difference in focus dZ is determined(S316). As described above, after the secondary subject is focused, thefocus is moved toward the main subject and at this time, the movingspeed is set to a value in accordance with the difference in focus dZ.Here, the focus is moved from the secondary subject to the main subjectin a fixed time regardless of the difference in focus dZ. As a fixedtime, for example, about three seconds are set, however, this may bechanged appropriately in consideration of the preferences of aphotographer and a viewer.

Subsequently, an image is captured and recorded while eliminating thedifference in exposure dE (S317). In step S308, the difference inexposure dE is found and exposure is controlled so that an appropriateexposure is maintained in consideration of the difference in exposure dEduring the focus shift. In order to control the exposure, the aperturevalue, the shutter speed, the ISO sensitivity, etc., are controlled. Atthis time, while the image data from the imaging section 2 is beingsubjected to image processing for a motion picture in the imageprocessing section 5, the image data is recorded in the recordingsection 6 as a motion picture.

Next, whether or not the Za region is reached is determined (S318). TheZa region is a region where the main subject is present and whether ornot the focus position of the photographic lens reaches the region isdetermined. When the Za region is not reached according to the result ofthe determination, the process returns to step S317 and an image iscaptured and recorded while eliminating the difference in exposure dE.

On the other hand, when the Za region is reached according to the resultof the determination in step S317, a still image is captured andrecorded (S319). The Za region is the region of the main subject andwhen this region is reached, the image data from the imaging section 2is subjected to image processing for a still image by the imageprocessing section 5 and the image data of the still image is recorded.The state in step S319 corresponds to the image 25 b illustrated in FIG.25B and in this case, the state is one where the person 21 a is infocus. Because the exposure control is changed so that the difference inexposure dE is eliminated, an appropriate exposure is achieved also forthe still image. It may also be possible to shoot the still image afterthe movement of focus shift stops in the Za region. After the image dataof the still image is recorded, the art motion picture mode iscompleted.

In the flowchart shown in FIG. 29, when the release button is pressedfully according to the result of the determination in step S309,photographing is started after the focus is shifted to the Zb region.However, the focus is adjusted to the Zb region from the state where theZa region is in focus, and therefore, a time lag occurs corresponding tothe time required for focus adjustment. Hence, there is a possibilitythat photo opportunity is lost, and therefore, the level of emergency isdetermined based on the impact when the button is pressed fully etc.When the level of emergency is high, it may also be possible to performstill image photographing in the Za region first and then perform themotion picture photographing while shifting focus toward the Zb region.

As described above, in the fifth embodiment of the present invention,the screen is divided into a plurality of regions in accordance with thesubject and a region from which the focus is moved is determined fromthe plurality of regions and the focus is moved by the focus controlsection. Hence, it is possible to capture a varied image without theneed of a photographer to perform a manual operation intentionally.Further, the focus control section is configured to record image data ofa motion picture when moving the focus, and therefore, it is possible toeasily perform motion picture photographing.

Further, in the fifth embodiment of the present invention, the focuscontrol section performs motion picture photographing while moving thefocus and performs still image photographing when the target focusposition is reached. Hence, it is possible to easily capture a variedimage that fuses a motion picture and a still image together. The motionpicture obtained by such photographing exhibits a very significanteffect without the presence of the still image and will serve as aneffective raw material when only the motion picture parts are connectedand edited.

Furthermore, in the fifth embodiment of the present invention, when thesubject image includes the main subject and first and second image partsother than the main subject, the direction of the movement of the focusis determined using the difference in focus between the main subject andthe first image part and the difference in focus between the mainsubject and the second image part. Specifically, when determining thesecondary subject, the fact that the focus is apart from the position offocus adjustment (10) in FIG. 28 is taken into consideration. Hence, theeffect of focus shift turns out remarkably and a varied image isobtained.

In the fifth embodiment of the present invention, when information aboutdistance to subject is acquired, the contrast method is used, in whichthe high-frequency component of image data is extracted, however, thisis not limited and other ranging methods may be used. For example, adifference in phase AF may be employed and in this case, it is no longernecessary to scan the photographic lens.

Further, in the fifth embodiment of the present invention, when thescreen is divided, information about distance to subject is used,however, this is not limited and other methods may be used. For example,it may also be possible to analyze the image and extract the contour,then, divide the screen based on the contour.

Furthermore, in each of the embodiments of the present inventiondescribed above, explanation is given using a digital camera as a devicefor photographing, however, the camera may be a digital single-lensreflex camera or a compact digital camera, a camera for motion picturephotographing, such as a video camera and a movie camera, or a cameraincorporated in a mobile telephone, personal digital assist (PDA)terminal, gaming machine, etc.

The present invention is not limited to the above-mentioned embodimentsas they are but can be embodied by modifying components withoutdeviating from the scope of the gist in the embodiment stage. Further,it is also possible to form various inventions by appropriatelycombining the plurality of components disclosed in the above-mentionedembodiments. For example, it may also be possible to delete somecomponents of the entire components shown in the embodiments.Furthermore, it may also be possible to appropriately combine thecomponents across the different embodiments.

What is claimed is:
 1. Apparatus comprising: a control section that (1)receives image data including a still image and a plurality ofsequentially captured images, (2) automatically trims a part of imagefrom at least one of the sequentially captured images to extract aregion corresponding to a part of the still image, and (3) generates atrimmed image from the extracted region; and a recording section thatrecords the still image and the trimmed image.
 2. The apparatusaccording to claim 1, wherein the region extracted corresponds to acenter of the still image.
 3. The apparatus according to claim 1,wherein the region extracted corresponds to a face part of the stillimage.
 4. The apparatus according to claim 1, wherein an aspect ratio ofat least one of the trimmed images is different from an aspect ratio ofthe still image.
 5. A method comprising: receiving image data includinga still image and a plurality of sequentially captured images;automatically trimming a part of image from at least one of thesequentially captured images to extract a region corresponding to a partof the still image; generating a trimmed image from the extractedregion; and recording, on a non-transitory storage medium, the stillimage and the trimmed image.